CS4706 Spring 2011 - Project 1 - "Building a TTS System"

Submission procedure explained below.

In this assignment, you are going to build your own limited-domain text-to-speech system. This is a complex task, so it is divided into three simpler subtasks.

Part A: Setup and first small TTS system

IMPORTANT: This part of the assignment cannot be done remotely. You have to do it on one of the Linux computers in the Speech Lab, for which you need to sign-up first.

Note: It might also be possible to do this locally on one of the Linux computers in the CLIC Lab. However, we have not tested the software there, and we cannot guarantee that it will work. So if you do this you will be on your own."

 

A.1. Setup

Add these lines to your ~/.bashrc file:

export PATH=/proj/speech/tools/festival/bin:$PATH
export PATH=/proj/speech/tools/festival/speech_tools/bin:$PATH
export FESTVOXDIR=/proj/speech/tools/festival/festvox
export ESTDIR=/proj/speech/tools/festival/speech_tools
 

Remember that you have to log out and back in for the changes to the ~/.profile file to take effect. You can check if the changes are in effect by running echo $FESTVOXDIR. The value specified above should be displayed.

A.2. Small TTS system: a talking clock

Log on locally to a Linux computer in the Speech Lab. Open a Terminal window (Applications » Accessories » Terminal) and run the following commands, replacing USERNAME with the user name of your CS account (e.g. fb2175).

In http://festvox.org/bsv/x1003.html you can find a detailed explanation of each step.


 
Step Commands Comments
1 mkdir /proj/speech/users/cs4706/USERNAME
cd /proj/speech/users/cs4706/USERNAME
mkdir time
cd time
Create a directory and cd into it.
2 $FESTVOXDIR/src/ldom/setup_ldom SLP time xyz Setup dir
  At this point, take a look at these two files:
etc/time.data, which contains a set of utterances that should cover all the possible variations in the domain;
festvox/SLP_time_xyz.scm, which defines several functions (in Scheme) to convert a time like "07:57" into an utterance like "The time is now, a little after five to eight, in the morning".
In order to build a new limited domain it is necessary to rewrite these files. For this part of the homework you do not need to edit these files, but you will in Parts B and C.
3 festival -b festvox/build_ldom.scm '(build_prompts "etc/time.data")' Generate prompts
4 bin/prompt_them etc/time.data Record prompts.
You need a microphone for this step. You will be asked to read out 24 prompts. Read the recording tips before starting.
5 bin/make_labs prompt-wav/*.wav Autolabel prompts
6 festival -b festvox/build_ldom.scm '(build_utts "etc/time.data")' Build utterances
7 cp etc/time.data etc/txt.done.data  
8 bin/make_pm_wave wav/*.wav
bin/make_pm_fix pm/*.pm
Extract pitchmarks & fix them
9 bin/simple_powernormalize wav/*.wav Power normalization
10 bin/make_mcep wav/*.wav MCEP vectors
11 festival -b festvox/build_ldom.scm '(build_clunits "etc/time.data")' Build LDOM Synthesizer
12 festival festvox/SLP_time_xyz_ldom.scm '(voice_SLP_time_xyz_ldom)' Run your synthesizer
13 (saytime)
(saythistime "07:57")
(saythistime "14:22")
Once in Festival, use these commands to make your synthesizer say the time.
Use CTRL+D to exit Festival.

Documentation

Requirements:

  1. To grade your work, we need to run your TTS system. Therefore, please double check that all your files under /proj/speech/users/cs4706/USERNAME have read permissions for everybody, and also exec permissions in the case of directories (that is the default setting, so if you have not chmoded anything, you do not need to change anything).
  2. You should generate three wav files using your TTS with the following procedure:

    cd /proj/speech/users/cs4706/USERNAME/time
    festival

    (load "festvox/SLP_time_xyz_ldom.scm")
    (voice_SLP_time_xyz_ldom)

    (Parameter.set 'Audio_Method 'Audio_Command)
    (Parameter.set 'Audio_Required_Rate 16000)
    (Parameter.set 'Audio_Required_Format 'wav)

    (Parameter.set 'Audio_Command "cp $FILE time1.wav")
    (saytime)

    (Parameter.set 'Audio_Command "cp $FILE time2.wav")
    (saythistime "07:57")

    (Parameter.set 'Audio_Command "cp $FILE time3.wav")
    (saythistime "14:22")
     


 

Part B: Preparing your limited-domain TTS

 

IMPORTANT: All steps in Part B can be done remotely.

B.1. Choosing a limited domain:

In Part A, you built a simple talking-clock TTS system. Now, you will build a TTS system for the Spoken Dialogue System application you have chosen. And, instead of recording it using a neutral voice, you may want to choose a particular style or personality that you think is most appropriate for your domain and application.

B.2. Designing the input and output of your TTS system

Define as formally as possible what the input and output of your TTS system is going to look like.

Your limited domain must have at least three degrees of freedom, and you have to provide an estimate of the number of possible sentences that could be generated.

B.3. Setting things up

Log on to clic.cs.columbia.edu with your CS account (either locally or remotely), and run the following commands, where USERNAME is the user name of your CS account (e.g. fb2175), and TOPIC is a string such as 'number', 'weather', 'street', etc.:

Step Commands Comments
1 mkdir /proj/speech/users/cs4706/USERNAME/partc
 
Create a directory for part c.
2 cd /proj/speech/users/cs4706/USERNAME/partc
$FESTVOXDIR/src/ldom/setup_ldom SLP TOPIC xyz
Setup the directory for part c.

B.4. Designing the prompts

Next, you need to design the prompts for your TTS system. As you saw in Part A, the talking clock uses the prompts in time/etc/time.data:

( time0001 "The time is now, exactly five past one, in the morning." )
( time0002 "The time is now, just after ten past two, in the morning." )
( time0003 "The time is now, a little after quarter past three, in the morning." )
...

Now, you have to create a similar file for your domain, and save it as partc/etc/TOPIC.data

For an explanation on how to design the prompts, go to http://www.festvox.org/bsv/c941.html#AEN952.

Documentation


 

Part C: Completing your limited-domain TTS

IMPORTANT: Section C.2 of this part of the work cannot be done remotelly. You have to do it on one of the Linux computers in the Speech Lab, for which you need to sign-up first.

Note: It might also be possible to do this work locally on one of the Linux computers in the CLIC Lab. However, we have not tested the software there, and we cannot guarantee that it will work. So if you do this you will be on your own.

 

C.1. Introduction

In Part B, you started building your limited-domain TTS system. You defined its input and output, designed the set of prompts you will record.

Now, in order to complete your TTS system you need to:
a) record the set of prompts (section C.2);
b) write a script that transforms an input string into an English sentence, and sends it to Festival to synthesize it (section C.3).

C.2. Recording and processing the prompts

Log on locally to a Linux computer in the Speech Lab. Open a Terminal window (Applications » Accessories » Terminal) and run the following commands, replacing USERNAME with the user name of your CS account (e.g. fb2175), and TOPIC with a string such as 'number', 'weather', 'street', etc.

Before starting, read the tips for part C, which you may find useful.

Step Commands Comments
1 cd /proj/speech/users/cs4706/USERNAME/partc  
2 The file etc/TOPIC.data should contain the prompts you designed. Make sure that its syntax is correct.
3 festival -b festvox/build_ldom.scm '(build_prompts "etc/TOPIC.data")' Generate prompts
4 bin/prompt_them etc/TOPIC.data Record prompts.
You need a microphone for this step. You will be asked to read out, one by one, the prompts you designed. Before starting, review the recording tips.
5 bin/make_labs prompt-wav/*.wav Autolabel prompts
6 festival -b festvox/build_ldom.scm '(build_utts "etc/TOPIC.data")' Build utterances
7 cp etc/TOPIC.data etc/txt.done.data  
8 bin/make_pm_wave wav/*.wav
bin/make_pm_fix pm/*.pm
Extract pitchmarks & fix them
9 bin/simple_powernormalize wav/*.wav Power normalization
10 bin/make_mcep wav/*.wav MCEP vectors
11 festival -b festvox/build_ldom.scm '(build_clunits "etc/TOPIC.data")' Build LDOM Synthesizer
12 festival festvox/SLP_TOPIC_xyz_ldom.scm '(voice_SLP_TOPIC_xyz_ldom)' Run Festival (use CTRL+D to exit).
13 Now, you can make your synthesizer say sentences in your domain by hand. For example, in the time domain you could do that by running: (SayText "The time is now, a little after twenty past two, in the afternoon.")
 

Warning: If you use words that do not belong to your domain, the synthesizer will fail.
 

C.3. Writing a script that transforms the input and synthesizes it

Now, you need to write a script that receives an input string as you defined in Part B, and transforms it into an English sentence that your TTS system can synthesize. For example, in the time domain the script would transform a string like "14:22" into a sentence like "The time is now, a little after twenty past two, in the afternoon." A simple (and perhaps long) case or switch statement should be enough to achieve this.
You must test your script properly before submission (in one of the speech-lab machines). If you sibmit a not working script, you will lose a lot of points, and we will not debug your code.

Use this Perl script as a basis. Update the variables $USERNAME and $TOPIC, complete the code where marked, and define the function generate_sentence, which does the input-output transformation. Please comment your code thoroughly.

The rest of the code in this script creates a temporary Festival script and runs it. That temporary Festival script loads your limited domain and creates a wav file with the resulting synthesis (you did exactly the same thing by hand in Part A for the time domain). You should not need to modify any of this.

Note: It is also possible to do the transformation part of the script (input string to English sentence) using the Scheme programming language, and later import your script into Festival. If you want to do it that way, please check with the TA first.

Documentation

Submission


Creare a folder YourUni-PROJ1 (e.g., fb2175-PROJ1) create three subdirectories, parta, partb, and partc -- YOU MUST FOLLOW THESE CONVENTIONS EXACTLY.

1. For Part A, save the three wave files: time1.wav, time2.wav, and time3.wav (see part A) in parta subfolder.

2. For Part B, save the following files in partb subfolder:


3. For Part C, save the following files in partc subfolder:
Now, compress (in zip format only) the main folder to YourUni-PROJ1.zip (e.g., fb2175-PROJ1.zip). Submit this zip file in Courseworks.

Tips for recording (for parts A and C only)

Further tips for part C