Light Field Transfer: Global Illumination Between Real and Synthetic Objects |
 |
Compositing real and synthetic objects in the same scene is important
for many computer graphics applications such as visual effects,
augmented reality, and architectural rendering. To make composite scenes compelling, lighting from local and distant sources
must be consistent between real and synthetic components. Moreover,
when synthetic and real objects are in close proximity to one
another, indirect lighting plays an important role in creating realistic
renderings. Most existing methods only account for distant lighting,
and do not consider interreflections between real and synthetic
scenes.
This project demonstrates a novel image-based method for compositing real and
synthetic objects in the same scene with a high degree of visual realism. Ours
is the first technique to allow global illumination and near-field lighting
effects between both real and synthetic objects at interactive rates, without
needing a geometric and material model of the real scene. We achieve this by
using a light field interface between real and synthetic components, simulating
indirect illumination using only two 4D light fields; one captured from and one
projected onto the real scene. Multiple bounces of interreflections are
obtained by iterating this approach. The interactivity of our technique enables
its use with time-varying scenes, including dynamic objects. This is in sharp
contrast to the alternative approach of using 6D or 8D light transport
functions of real objects, which are very expensive in terms of acquisition and
storage and hence not suitable for real-time applications. In our method, 4D
radiance fields are simultaneously captured and projected by using a lens
array, video camera, and digital projector. The method supports full global
illumination with restricted object placement, and accommodates moderately
specular materials. We implement a complete system and show several example
scene compositions that demonstrate global illumination effects between dynamic
real and synthetic objects. Our implementation requires a single point light
source and dark background.
|
Publications
"Light Field Transfer: Global Illumination Between Real and Synthetic Objects," O. Cossairt, S. K. Nayar, and R. Ramamoorthi, ACM Trans. on Graphics (also Proc. of ACM SIGGRAPH), Aug. 2008. [PDF] [bib] [©]
|
Images
 |
|
Conceptual Diagram:
This diagram shows a high-level description of the dataflow for light field
transfer. In our technique, indirect lighting is transferred between real and
virtual scenes via a light field interface.
|
|
|
|
 |
|
Hardware Layout:
The system consists of a light field capture and projection unit, a view
camera, and direct lighting. A camera and projector share a common lens array
that multiplexes 2D radiance patterns into 4D patterns.
|
|
|
|
 |
|
Multiple Bounce Global Illumination:
Here we show the decomposition of lighting for a simple scene consisting of a
synthetic green cube and a real red block. The figure shows the contribution to
each object from direct, first bounce indirect, and second bounce indirect
illumination. The final result, including a sum of direct and indirect
contributions is also shown, along with an all-synthetic scene having similar
material properties that is rendered with path tracing for comparison and
verification.
|
|
|
|
 |
|
Glossy Reflections:
In this example we show glossy reflections of a synthetic photograph on a
shiny bowl in the real scene. As the frame moves closer, its reflection
enlarges and more detail is visible. This scene is also shown with a slightly
different configuration at the top of this page, where we also see glossy
reflections of the real hand, orange, and bust in the synthetic photo frame.
|
|
|
|
 |
|
Soft Shadows and Diffuse-Diffuse Interactions:
Here, the virtual scene consists of a sun and an orbiting planet. The real
scene consists of a mannequin head. There is no direct lighting in this scene,
and the only illumination of the mannequin comes from the synthetic sun. The
synthetic planet casts a shadow on the mannequin, and some light reflected off
the mannequin illuminates the shadowed side of the planet. As the planet moves
closer to the mannequin, the soft shadow becomes more distinct, and
illumination from the mannequin on the planet increases.
|
|
|
|
|
Videos
 |
|
Siggraph 2008 Video:
This video includes a detailed description of the lightfield transfer method,
footage of the hardware used, as well as example videos demonstrating real-time
applications. The examples are taken from the same scenes as in the pictures
section above. (With narration)
|
|
|
|
 |
|
Picture Frame and Shiny Bowl:
This video demonstrates a dynamic scene with glossy reflections shared between
real and synthetic objects. The footage was generated in near real-time using
our Light Field Transfer method. The sequence uses the same scene as in the
'Glossy Reflections' example above.
|
|
|
|
 |
|
Planets and Mannequin:
This video demonstrates a dynamic scene with soft shadows and diffuse-diffuse
interactions shared between real and synthetic objects. Again, the footage was
generated in in near real-time. The sequence uses the same scene as in the
'Soft Shadows and Diffuse-Diffuse Interactions' example above.
|
|
|
|
|
Lighting Sensitive Display
Project Anywhere: Radiometric and Geometric Compensation
Non-Single Viewpoint Imaging: Raxels and Caustics
|
|
|