Optimal Illumination for Video Relighting

In this project, we are interested in capturing a video of a possibly dynamic scene using a set of repeating basis illuminations and then relighting the captured video with a novel and possibly varying illumination. To achieve high quality relighting of dynamic scenes, we need to ensure that the number of basis illuminations used are small. In short, we wish to determine the optimal lighting basis (OLB) to illuminate a given scene. Although the OLB depends on the scene, we show that it can be computed via a simple calibration procedure that is done before the video of interest is captured.

We demonstrate through experiments on real and synthetic data that the OLB significantly reduces the number of basis illuminations that are needed to achieve a desired level of accuracy in the relit video. Since the number of basis illuminations is small, only a few neighboring video frames need to be aligned to relight each frame. We present several relighting results for scenes with moving objects, including faces. In each case, although a single video clip was captured, we are able to relight the scene with novel and even varying lighting conditions.

Publications

"Optimal Illumination for Image and Video Relighting,"
F. Moreno-Noguer, S.K. Nayar and P.N. Belhumeur,
IEE European Conference on Visual Media Production (CVMP),
pp.199-208, Dec, 2005.
[PDF] [bib] [©]

"Optimal Illumination for Image and Video Relighting,"
F. Moreno-Noguer, S.K. Nayar and P.N. Belhumeur,
SIGGRAPH 2005 Technical Sketch,
pp.00,
Aug, 2005.
[PDF] [bib] [©]

Image

  Comparison of OLB and other Lighting Bases:

Some reconstructed images (and their corresponding errors) for synthetic static objects which have been illuminated by Spherical Harmonic Lighting Basis (SHLB), Fourier Lighting Basis (FLB) and the proposed Optimal Lighting Basis (OLB). Note how the OLB performs significantly better.

     

Video

If you are having trouble viewing these .mpg videos in your browser, please save them to your computer first (by right-clicking and choosing "Save Target As..."), and then open them.

  SIGGRAPH 2005 Sketch Video:

This video shows how the optimal lighting basis is computed and several examples of video relighting. (With narration)

     

Slides

EGSR 2007 presentation     With videos (zip file)

Multiplexed Illumination