Bob Coyne Visualizing the
meaning of language
Home
Research
Bibliography
We live in a vast sea of ever-changing text with few tools available to help us visualize its meaning. The goal of this research is to bridge the gap between graphics and language by developing new theoretical models and supporting technology to create a system that automatically converts descriptive text into rendered 3D scenes representing the meaning of that text. This builds upon previous work done with Richard Sproat in the WordsEye text-to-scene system (available online at www.wordseye.com). New research areas include:

Contextual Reasoning: Scenes are often described with oblique contextual references to background settings and ongoing actions. Similarly, properties of a scene's constituent objects can constrain the interpretation of the given text. These and other contextual cues can help resolve ambiguities and build rich, robust models of depicted scenes.

Lexical Semantic Framework: Our goal is to develop a new representation of lexical meaning which builds on but goes beyond existing frameworks (such as FrameNet's rich representation of verb frames) by incorporating new lexial semantic relations that refer to contextual knowledge.

Storyboards: Textual descriptions often include shifts in time and location as well as other changes of state. The goal of the system is to recognize such changes and depict them in storyboard fashion. This mimics human cognition, where people may not fill in all details, but mentally skip between salient clusters.

Knowledge Acquisition: Given the ability to perform basic syntactic and semantic analysis of text and an understanding of the selectional restrictions imposed by context and object properties, the system will acquire scene-related world knowledge from large corpora.
 Examples from previous research
Input text:  A silver head of time is on the grassy ground. The blossom is next to the head. It is in the ground. The green light is three feet above the blossom. The yellow light is 3 feet above the head. The large wasp is behind the blossom. It wasp is facing the head. Input text:  The brick wall is 120 feet wide and 4 feet tall. The wall is behind the willow tree. The tree is on the mountain range. the mountain range has a dirt texture. The church is 10 feet behind the wall. The shiny sphere is 40 feet above the ground and 10 feet to the right of the church. The sphere is 30 feet tall. The huge silver cowboy is 12 feet behind the church.