Spoken Language Processing (CS 4706), Spring 2006


MW  2:40-3:55




Julia Hirschberg

Office Hours: 

Wed 4:00-6:00, CEPSR 705


julia [at] cs.columbia.edu



Teaching Assistant:

Agus Gravano

Office Hours:

Mon 1:30-2:30, CEPSR 726
Tue 3:30-4:30, Speech Lab


agus [at] cs.columbia.edu



Announcements || Academic Integrity || Description
Resources || Requirements || Syllabus || Readings


This course introduces students to research in spoken language in computational linguistics, aka natural language processing (NLP). We will study the different `meanings' that can be conveyed by the way that speakers produce sentences, techniques for analyzing spoken language, methods of developing speech technologies, and applications of such technologies in the real world, such as text-to-speech systems, speech recognizers, spoken dialogue systems, and detectors for various types of emotional speech.  NB: This course can be counted as a PhD elective in Advanced AI .  It is a requirement for the MS NLP Track.  There are no official prerequisites for this course except Data Structures or equivalent, and no prior knowledge of NLP will be assumed.


Required readings: Acoustic & Auditory Phonetics by Keith Johnson, and selected chapters from Speech and Language Processing by Jurafsky and Martin (check out the errata for the first edition). Both are available from Morningside Bookstore and will also be on reserve in the library.  Other course readings will be available either on the web or in-class handouts.  Note that ‘*’ indicates an optional reading.

Course Requirements:

Midterm and final; 4 lab homeworks; a 5-page critical essay (see below). The Speech Lab is available for use in homeworks on a signup basis.

40% Homeworks
10% Critical essay
20% Midterm Exam
30% Final Exam

Late policy:
Each student starts the semester with 5 late days. Students use up late days when they turn in homework anytime after the due date and time.  For example, if homework is due at 2:40 pm on Wednesday, anything turned in after 2:40 pm on Wednesday, but before 2:40 pm on Thursday uses up one late day. Once the ‘free’ late days are exhausted, homework submitted after the due date will be penalized @10% per late day (e.g. 3 days late, grade will be penalized by 30%).
Late days can be used for both the homeworks and the critical essay.
(Note:  Weekdays and weekends all count equally in the late day calculation.)

Critical Essay:
• It should be about the full set of readings for any lecture topic with multiple papers assigned.
• It should include regular and optional articles.
• You should compare and contrast each of the articles, expressing your opinion on what you think is good or bad about them.
• It should be at least 5 pages.

Submit one good question about the assigned reading before the start of each lecture, by e-mail to agus [at] cs.columbia.edu and with the exact subject "CS4706 Question".

Academic Integrity:

Copying or paraphrasing someone's work (code included), or permitting your own work to be copied or paraphrased, even if only in part, is not allowed, and will result in an automatic grade of 0 for the entire assignment or exam in which the copying or paraphrasing was done. Your grade should reflect your own work. If you believe you are going to have trouble completing an assignment, please talk to the professor in advance of the due date.



Praat - Praat resources


Help using ToBI - ToBI Annotation Environments

Text-to-Speech Links and more...

Text-to-Song synthesis





Readings and Assignments

Reports and HW


W Jan 18

Introduction to the Course




M Jan 23

It's not what you said, it's how you said it

Hirschberg03 [ps] [pdf]



W Jan 25

Acoustics of Speech

Johnson, Ch. 1-2



M Jan 30

Tools for Speech Analysis

Praat tutorial

HW1: Using Praat (assigned)


W Feb 1

From Sounds to Language

J&M 4-4.2



M Feb 6

Intonational Variation




W Feb 8

 ToBI and ToBI Labeling

 ToBI labeling conventions

Listen to the ToBI examples

HW1 due
HW2: ToBI (assigned)


M Feb 13

Studying Intonation:  How do people ask questions?

Wilson93;Hedberg&Sosa02; Syrdal&Jilka04; *Dohertyetal04



W Feb 15

Speech Generation

HLT96-ch5, TTS systems


M Feb 20

Modeling Pronunciation

J&M, 4.6; Fackrell&Skut04

HW2 due
HW3a: TTS (assigned)


W Feb 22

Predicting Accents and Phrasing

J&M, 4.7
Pan99, *Sun02, Koehn00, Rambow01

HW1: Good thermometers!


M Feb 27

Information Status: Focus and Given/New

Nakatani99, GBrown83, *Bard99, Prince92, Dahan02




W Mar 1

Discourse Structure in TTS Systems

J&M, 18.3, 19.4

HW3a due
HW3b assigned

HW2: bdc-solved.TextGrid games-solved.TextGrid


M Mar 6

Midterm Exam




M Mar 8

Speech Recognition and Understanding

*J&M, 7.1-7.8 (new version is here); HLT96-ch1



M Mar 20

Speech Disfluencies




W Mar 22

Sentence, Speaker, and Topic Segmentation

Shriberg00, Choi00, *Utiyama01,

HW3b due
HW3c assigned


M Mar 27

Identifying Speech Acts

J&M, 19.5-6 (new version)
*Jurafsky98, Rosset&Lamel04, Nickerson&Chu-Carroll99



W Mar 29

Spoken Dialogue Systems

J&M, 19.1-3,9 (new version)
Walkeretal97, *Bell&Gustafson00



M Apr 3

Turntaking in Spoken Dialogue

J&M, 19.7 (new version)
Conversational Analysis and Turn-Taking



W Apr 5

Dialogue Errors and Corrections

J&M, 19.4 (new version)
Hirschbergetal04, Goldberg03, Krahmer01



M Apr 10

Speech Data Mining

Maskeyetal04, Hearst99

HW3c due


W Apr 12

Speech Summarization

*Koumpis&Renals05, Hori02, Zechner01



M Apr 17

Emotional Speech

Cowie00, *Pereira00, Schroeder01, *Bosch00, Burkhardt00, Ang02 ,*Gobl&Chasaide03

HW4: Perception Test (assigned)


W Apr 19

Deceptive Speech

 DePauloetal83, Frank92, *Mehrabian77, Streeteretal71



M Apr 24

Charismatic Speech

Boss76, Tuppen74, Weber47

HW4 due


W Apr 26

Summing Up




M May 1

No Class


Critical Essays due


May 2-4

Study Days




May 5-12

Final Exams



Announcements || Academic Integrity || Description
Links to Resources || Requirements || Syllabus || Text || Thanks