CS4706 - Spring 2006
Homework 3 - "Building a TTS System"

Part A: Setup and first small TTS system

Due: Wednesday, March 1, 2006, by 2:40pm.
Submission procedure explained below.

A.1. Setup

Add these lines to your ~/.profile file:

export PATH=$PATH:/proj/speech/users/cs4706/speech/festival/bin/
export PATH=$PATH:/proj/speech/users/cs4706/speech/speech_tools/bin/
export FESTVOXDIR=/proj/speech/users/cs4706/speech/festvox
export ESTDIR=/proj/speech/users/cs4706/speech/speech_tools

Remember that you have to log out and back in for the changes to the ~/.profile file to take effect. You can check if the changes are in effect by running echo $FESTVOXDIR. The value specified above should be displayed.

A.2. Small TTS system: a talking clock

Log on locally to a Linux computer in the Speech Lab. Start X-windows with the command startx after logging in. Once on X-windows, open a Terminal window (Start menu » System Tools » Terminal) and run the following commands, replacing USERNAME with the user name of your CS account (e.g. jfk2101).

In http://festvox.org/bsv/x1003.html you can find a detailed explanation of each step.

Documentation

• » Festival & FestVox home page
• » Building Synthetic Voices

Submission

1. To grade your homework, we need to run your TTS system. Therefore, please double check that all your files under /proj/speech/users/cs4706/USERNAME have read permissions for everybody, and also exec permissions in the case of directories (that is the default setting, so if you have not chmoded anything, you do not need to change anything).

2. Submit by e-mail to agus [at] cs.columbia.edu the three wav files generated by your TTS with the following procedure:

   cd /proj/speech/users/cs4706/USERNAME/time
   festival
   
   (load "festvox/SLP_time_xyz_ldom.scm")
   (voice_SLP_time_xyz_ldom)
   
   (Parameter.set 'Audio_Method 'Audio_Command)
   (Parameter.set 'Audio_Required_Rate 16000)
   (Parameter.set 'Audio_Required_Format 'wav)
   
   (Parameter.set 'Audio_Command "cp $FILE time1.wav")
   (saytime)
   
   (Parameter.set 'Audio_Command "cp $FILE time2.wav")
   (saythistime "07:57")
   
   (Parameter.set 'Audio_Command "cp $FILE time3.wav")
   (saythistime "14:22")
   

Part B: Preparing your limited-domain TTS

Due: Wednesday, March 22, 2006, by 2:40pm.
Submission procedure explained below.

B.1. Choosing a limited domain and two speaking styles

In Part A, you built a simple talking-clock TTS system. Now, you will build a TTS system with a limited domain of your choice. And, instead of recording it using a neutral voice, you will record two versions of the same system, each with a distinct style. You will also choose the manner in which the two speaking styles will differ.

So, first of all, choose a domain from the following list:

• Phone numbers
• Dates
• Weather report
• Street addresses
• Stock market quotes
• Other (ask first)

Next, choose the manner in which the two speaking styles will differ:

• Each style will have a particular personality (e.g., confident and shy).
• Each style will address a particular audience (e.g., children and adults).
• Other (ask first).

Choose carefully how the styles will differ, taking into account that you will have to record them using your own voice. Think how you will ellicit them in order to make them clearly distinguishable from each other.

B.2. Designing the input and output of your TTS system

Define as formally as possible what the input and output of your TTS system is going to look like.

• For example, in the talking clock the input is a string of the form HH:MM, and the output is a sentence of the form:

  The time is now, EXACTNESS MINUTE INFO(, in the DAYPART).

  where:

  EXACTNESS = {exactly, just after, a little after, almost}
  MINUTE = {-, five past, ten past, quarter past, twenty past, twenty-five past, half past, twenty-five to, twenty to, quarter to, ten to, five to}
  INFO = {one, two, three, four, five, six, seven, eight, nine, ten, eleven, twelve, midnight}
  DAYPART = {morning, afternoon, evening}

  Every time the talking clock receives an input like 07:57, it will translate it into a sentence like The time is now, a little after five to eight, in the morning, and later synthesize it.

  This TTS has four degrees of freedom (EXACTNESS, MINUTE, INFO, DAYPART). The number of possible sentences is approximately 4 x 12 x 12 x 2 = 1152.

Your limited domain must have at least three degrees of freedom, and you have to provide an estimate of the number of possible sentences that could be generated.

For each of the two speaking styles you will create a full instance of a Festival TTS system. They will be named style1 and style2, and they will share the limited domain, as well as the definition of input and output.

In Part C you will record each instance separately, adapting your own voice in order to convey the desired personality, or to address the desired audience.

B.3. Setting things up

Log on to clic.cs.columbia.edu with your CS account (either locally or remotely), and run the following commands, where USERNAME is the user name of your CS account (e.g. jfk2101), and TOPIC is a string such as 'number', 'weather', 'street', etc.:

B.4. Designing the prompts

Next, you need to design the prompts for your TTS system. As you saw in Part A, the talking clock uses the prompts in time/etc/time.data:

Now, you have to create a similar file for your domain, and save it twice, as style1/etc/TOPIC.data and style2/etc/TOPIC.data.
Both files should always be identical, so if you modify one of them do not forget to modify the other one too.

For an explanation on how to design the prompts, go to http://www.festvox.org/bsv/bsv-ldom-ch.html#AEN952.

Documentation

• » Festival & FestVox home page
• » Building Synthetic Voices
• » Designing the Prompts

Submission

Submit by e-mail to agus [at] cs.columbia.edu the following files:

• A text document specifying:
  1. a description of your limited domain;
  2. the two speaking styles you have chosen and how they will differ;
  3. the definition of the input and output of your TTS system, specifying the degrees of freedom and the number of possible sentences that could be generated.
• The file TOPIC.data which contains the prompts you have designed.

Part C: Completing your limited-domain TTS

Due: Monday, April 10, 2006, by 2:40pm.
Submission procedure explained below.

C.1. Introduction

In Part B, you started building your limited-domain TTS system. You defined its input and output, designed the set of prompts you will record, and decided how the two speaking styles will differ.

Now, in order to complete your TTS system you need to:
a) record the set of prompts, once with each speaking style (section C.2);
b) write a script that transforms an input string into an English sentence, and sends it to Festival to synthesize it (section C.3).

C.2. Recording and processing the prompts

Log on locally to a Linux computer in the Speech Lab. Start X-windows with the command startx after logging in. Once on X-windows, open a Terminal window (Start menu » System Tools » Terminal) and run the following commands, replacing USERNAME with the user name of your CS account (e.g. jfk2101), and TOPIC with a string such as 'number', 'weather', 'street', etc.

Before starting, read the tips for part C, which you may find useful.

Now repeat steps 1 to 12, this time using style2 instead of style1 in the first step. Be very careful with this, or you may overwrite your previous recordings!

C.3. Writing a script that transforms the input and synthesizes it

Now, you need to write a script that receives an input string as you defined in Part B, and transforms it into an English sentence that your TTS system can synthesize. For example, in the time domain the script would transform a string like "14:22" into a sentence like "The time is now, a little after twenty past two, in the afternoon." A simple (and perhaps long) case or switch statement should be enough to achieve this.

Use this Perl script as a basis. Update the variables $USERNAME and $TOPIC, complete the code where marked, and define the function generate_sentence, which does the input-output transformation. Please comment your code thoroughly.

The rest of the code in this script creates a temporary Festival script and runs it. That temporary Festival script loads your limited domain with the selected speaking style, and creates a wav file with the resulting synthesis (you did exactly the same thing by hand in Part A for the time domain). You should not need to modify any of this.

Note: It is also possible to do the transformation part of the script (input string to English sentence) using the Scheme programming language, and later import your script into Festival. If you want to do it that way, please check with Agus first.

Documentation

• » Festival & FestVox home page
• » Building Synthetic Voices

Submission

1. To grade your homework, we need to run your TTS system. Therefore, please double check that all your files under /proj/speech/users/cs4706/USERNAME have read permissions for everybody, and also exec permissions in the case of directories and scripts.
2. Submit by e-mail to agus [at] cs.columbia.edu the following files:
   • A plain text document (not Word, PDF, etc.) specifying:
     1. path and name of the Perl script you did in section C.3;
     2. a short description (2-3 sentences) of how you managed to convey the two speaking styles you chose;
     3. a description of any changes you made to your design in Part B, and an explanation of why you think they were necessary.
   • Do not submit any wav files.
     
   • A shell script sample.sh that generates three wav files (from three different inputs) for each speaking style. That is, given three inputs a, b and c, this script should generate two wav files for input a, two for input b, and two for input c. Six wav files in total.
     Choose the inputs that you consider show your system to its best advantage.
     The script should look like this:
     
     #!/bin/bash
     
     # path and name of the tts script (**update**)
     SCRIPTDIR=/proj/speech/users/cs4706/USERNAME
     SCRIPTNAME=tts.pl
     
     # the resulting wav files are saved in the present directory
     DESTDIR=`pwd`
     
     # generate two wav files for input #a
     $SCRIPTDIR/$SCRIPTNAME "03:39" 1 $DESTDIR/utt1a.wav
     $SCRIPTDIR/$SCRIPTNAME "03:39" 2 $DESTDIR/utt2a.wav
     
     # generate two wav files for input #b
     $SCRIPTDIR/$SCRIPTNAME "14:47" 1 $DESTDIR/utt1b.wav
     $SCRIPTDIR/$SCRIPTNAME "14:47" 2 $DESTDIR/utt2b.wav
     
     # generate two wav files for input #c
     $SCRIPTDIR/$SCRIPTNAME "23:55" 1 $DESTDIR/utt1c.wav
     $SCRIPTDIR/$SCRIPTNAME "23:55" 2 $DESTDIR/utt2c.wav
     

A complete example: cards

As an illustration, in /proj/speech/users/cs4706/speech/tts-cards there is a full example of a TTS system (with only one speaking style), embedded in a Java application.

1. cards/ contains all Festival and Festvox files.
2. cards/etc/cards.data is the file with the prompts designed for this particular domain.
3. CardToSpeech.pl is a Perl script similar to the one you have to prepare in Section C.3. Run it with no arguments to find out its usage and its input format. Make sure to enter the absolute path of the wav file (somewhere you have write permissions).
4. Run locally from a terminal on X-windows: java CardToSpeech
   This is a Java application that uses the previous script to synthesize description of cards. Have fun!

Tips for recording (for parts A and C only)

• Do not start without testing the microphone! To test it, try:
  rec -s w test.wav
  play test.wav
  Change the settings in the Volume Control (Start menu » Sound & Video » Volume Control) until your recorded speech is clear and loud enough, with as little background noise as possible.
• Try to make the recordings in a silent environment.
• Make sure that your microphone is not directly in front of your mouth, to avoid noise produced by your breath.
• The script bin/prompt_them in step 4 only lets us record all prompts in one sitting. If you make mistakes and want to record again a few specific prompts, you do not need to run that script again. Instead, follow these instructions (from the shell, not in Festival):
  
  1. Let PROMPT be the id of the prompt you want to record again (e.g. time0001).
  2. Run the command $ESTDIR/bin/ch_wave -info wav/PROMPT.wav
  3. Let DURATION be the duration in seconds of the wav file, which is given in the first line of the output.
  4. Run the command $ESTDIR/bin/na_record -f 16000 -time DURATION -o wav/PROMPT.wav -otype riff
     This will start recording immediately for the specified amount of seconds, and will replace the file wav/PROMPT.wav with the new recording.

Further tips for part C

• (This is very specific, most probably you will not need it.)
  If you want to ellicit the same word (or words) with a different intonation depending on its position in the sentence, you will find that Festival does not take word position into account, with one exception: a word immediately before a pause (comma, period) will be pronounced differently. For example, in "Pin number, one one one." Festival differentiates between the first two and the third "one", but not between the first and the second.
  Therefore, we have to find a way to change the pronunciation of a word, depending on its position.
  Suppose we have this set of prompts:
  ( pin0001 "Pin number, one twoh threehh." )
  ( pin0002 "Pin number, two threeh onehh." )
  ( pin0003 "Pin number, three oneh twohh." )
  Numbers in the second position have one extra h, and numbers in the third position have two extra h's.
  This lets us ellicit each number differently, depending on its position. The h's might mess up the synthesizer that reads out the sentences before recording, but do not seem to affect the rest of the process.
  Then, to access the different word instances when synthesizing speech, we have to use the correct version (e.g. one or oneh or onehh). Our prompts to the TTS system will look like:
  ( SayText "Pin number, two twoh onehh" )
  ( SayText "Pin number, three threeh twohh" )
  etc.