Light Field Transfer: Global Illumination Between Real and Synthetic Objects

Compositing real and synthetic objects in the same scene is important for many computer graphics applications such as visual effects, augmented reality, and architectural rendering. To make composite scenes compelling, lighting from local and distant sources must be consistent between real and synthetic components. Moreover, when synthetic and real objects are in close proximity to one another, indirect lighting plays an important role in creating realistic renderings. Most existing methods only account for distant lighting, and do not consider interreflections between real and synthetic scenes.

This project demonstrates a novel image-based method for compositing real and synthetic objects in the same scene with a high degree of visual realism. Ours is the first technique to allow global illumination and near-field lighting effects between both real and synthetic objects at interactive rates, without needing a geometric and material model of the real scene. We achieve this by using a light field interface between real and synthetic components, simulating indirect illumination using only two 4D light fields; one captured from and one projected onto the real scene. Multiple bounces of interreflections are obtained by iterating this approach. The interactivity of our technique enables its use with time-varying scenes, including dynamic objects. This is in sharp contrast to the alternative approach of using 6D or 8D light transport functions of real objects, which are very expensive in terms of acquisition and storage and hence not suitable for real-time applications. In our method, 4D radiance fields are simultaneously captured and projected by using a lens array, video camera, and digital projector. The method supports full global illumination with restricted object placement, and accommodates moderately specular materials. We implement a complete system and show several example scene compositions that demonstrate global illumination effects between dynamic real and synthetic objects. Our implementation requires a single point light source and dark background.

Publications

"Light Field Transfer: Global Illumination Between Real and Synthetic Objects,"
O. Cossairt, S. K. Nayar, and R. Ramamoorthi,
ACM Trans. on Graphics (also Proc. of ACM SIGGRAPH),
Aug. 2008.
[PDF] [bib] [©]

Images

  Conceptual Diagram:

This diagram shows a high-level description of the dataflow for light field transfer. In our technique, indirect lighting is transferred between real and virtual scenes via a light field interface.
     
  Hardware Layout:

The system consists of a light field capture and projection unit, a view camera, and direct lighting. A camera and projector share a common lens array that multiplexes 2D radiance patterns into 4D patterns.
     
  Multiple Bounce Global Illumination:

Here we show the decomposition of lighting for a simple scene consisting of a synthetic green cube and a real red block. The figure shows the contribution to each object from direct, first bounce indirect, and second bounce indirect illumination. The final result, including a sum of direct and indirect contributions is also shown, along with an all-synthetic scene having similar material properties that is rendered with path tracing for comparison and verification.
     
  Glossy Reflections:

In this example we show glossy reflections of a synthetic photograph on a shiny bowl in the real scene. As the frame moves closer, its reflection enlarges and more detail is visible. This scene is also shown with a slightly different configuration at the top of this page, where we also see glossy reflections of the real hand, orange, and bust in the synthetic photo frame.
     
  Soft Shadows and Diffuse-Diffuse Interactions:

Here, the virtual scene consists of a sun and an orbiting planet. The real scene consists of a mannequin head. There is no direct lighting in this scene, and the only illumination of the mannequin comes from the synthetic sun. The synthetic planet casts a shadow on the mannequin, and some light reflected off the mannequin illuminates the shadowed side of the planet. As the planet moves closer to the mannequin, the soft shadow becomes more distinct, and illumination from the mannequin on the planet increases.
     

Videos

  Siggraph 2008 Video:

This video includes a detailed description of the lightfield transfer method, footage of the hardware used, as well as example videos demonstrating real-time applications. The examples are taken from the same scenes as in the pictures section above. (With narration)
     
  Picture Frame and Shiny Bowl:

This video demonstrates a dynamic scene with glossy reflections shared between real and synthetic objects. The footage was generated in near real-time using our Light Field Transfer method. The sequence uses the same scene as in the 'Glossy Reflections' example above.
     
  Planets and Mannequin:

This video demonstrates a dynamic scene with soft shadows and diffuse-diffuse interactions shared between real and synthetic objects. Again, the footage was generated in in near real-time. The sequence uses the same scene as in the 'Soft Shadows and Diffuse-Diffuse Interactions' example above.
     

Lighting Sensitive Display

Project Anywhere: Radiometric and Geometric Compensation

Non-Single Viewpoint Imaging: Raxels and Caustics