CS6998: Topics in Speech Processing: Computational Approaches to Emotional Speech, Fall 2009

Time: Mon 4:10-6:00
Place: Eng.Terr. 253

Professor Julia Hirschberg (Office Hours: Th 4-6)
julia@cs.columbia.edu, 212-939-7114

Teaching Assistant Bob Coyne (Office Hours TBA)

Announcements | Academic Integrity | Description
Readings | Resources | Requirements | Syllabus


This course introduces students to research on emotional speech and its applications in call centers, tutoring systems, and medical diagnosis. We will explore state-of-the-art work on recognizing and producing classic emotions automatically. Emotions such as anger, happiness, sadness, and uncertainty are important to recognize in online dialogue systems and to produce in computer games. We will also study the recognition of other types of speaker state, including deceptive and charismatic speech, and the uses of acoustic and prosodic information in medical domains, for the diagnosis of mental and physical conditions. Classes will be lecture and discussion with an emphasis on group participation. There are no prerequisites for the course and no exams.


Students will lead one class discussion, and prepare weekly discussion questions based on assigned readings, and prepare a term project. The Columbia Speech Lab is available for use in homeworks on a signup basis.

All students are required to have a Computer Science Account for this class. To sign up for one, go to the CRF website and then click on "Apply for an Account".


Academic Integrity

Copying or paraphrasing someone's work (code included), or permitting your own work to be copied or paraphrased, even if only in part, is not allowed, and will result in an automatic grade of 0 for the entire assignment or exam in which the copying or paraphrasing was done. Your grade should reflect your own work. If you believe you are going to have trouble completing an assignment, please talk to the instructor or TA in advance of the due date.


Required readings will be available on line.




Date Topic Readings Reports & HW
Sep 14
Introduction The Love Detector; Emotional Synthetic Speech

Sep 21
Evidence of Emotion: Theory, Experiments and Corpora
Wiltingetal06; Scherer2003; Enos&Hirschberg06;
Nisimuraelal06; Cornelius00

Sep 28
Cues to Emotion: Face, Body, Brain (Bob Coyne)
Johnstoneetal06,Ekman video;
Ekman bio;

Oct 5
Cues to Emotion: Language (Suzanne Yuen); Introduction to Praat (Fadi Biadsy) Gupta&Rajput07; cf DAL; Praat; Yanushevskayaetal05 Project Ideas/Description
Oct 12
Techniques for Emotion Classification (Kaushal Lahankar)
Kumaretal06; Satohetal07;
cf. Feeltrace

Oct 19
Cultural Differences (Rivka Levitan, Ankit Ranka)
Braun&Katerbow05; Shahidetal07;
Change in weekly assignments
Oct 26
Producing Emotional Speech (Gabe Schubiner)
Cahn90; Eideetal04;
cf Synface demo, Mary TTS

Nov 9
Emotion in IVR Systems:  Anger and Frustration (Yves Scherrer)

Nov 16
Emotion in Tutoring Systems: Confidence and Confusion (Satyajeet Shaligram)
Aietal06; Liscombeetal05b

Nov 23
Emotion in Meetings:  Hot Spots and Laughter (Sidharatha Nallu)
Wrede&Shriberg03; Laskowski&Burger07

Nov 30
Deception Detection (Arthi Ramachandran, Ravi Kiran)
Hirschbergetal05; Eriksson&Lacerda07

Dec 7
Medical Diagnosis (Promitti Dutta)
Luoetal06; Levitetal01; Kalioubyetal06; Zei_Pollermann00

Dec 14
Modeling Other Speaker State:  Sarcasm, Charisma, Urgent/Personal (Yang Wang)
Teppermanetal06; Biadsyetal08;

Dec 23

Projects due


Julia Hirshberg Portrait

Julia Hirschberg
Professor, Computer Science

Columbia University
Department of Computer Science
1214 Amsterdam Avenue
M/C 0401
450 CS Building
New York, NY 10027

email: julia@cs.columbia.edu
phone: (212) 939-7114

Download CV

Columbia University Department of Computer Science / Fu Foundation School of Engineering & Applied Science
450 Computer Science Building / 1214 Amsterdam Avenue, Mailcode: 0401 / New York, New York 10027-7003
Tel: 1.212.939.7000 / Fax: 1.212.666.0140