In this project, we are interested in capturing a video of a possibly dynamic scene using a set of repeating basis illuminations and then relighting the captured video with a novel and possibly varying illumination. To achieve high quality relighting of dynamic scenes, we need to ensure that the number of basis illuminations used are small. In short, we wish to determine the optimal lighting basis (OLB) to illuminate a given scene. Although the OLB depends on the scene, we show that it can be computed via a simple calibration procedure that is done before the video of interest is captured.

We demonstrate through experiments on real and synthetic data that the OLB significantly reduces the number of basis illuminations that are needed to achieve a desired level of accuracy in the relit video. Since the number of basis illuminations is small, only a few neighboring video frames need to be aligned to relight each frame. We present several relighting results for scenes with moving objects, including faces. In each case, although a single video clip was captured, we are able to relight the scene with novel and even varying lighting conditions.

Publications

"Optimal Illumination for Image and Video Relighting,"
F. Moreno-Noguer, S.K. Nayar and P.N. Belhumeur,
IEE European Conference on Visual Media Production (CVMP),
pp.199-208, Dec, 2005.
[PDF] [bib] [©]


"Optimal Illumination for Image and Video Relighting,"
F. Moreno-Noguer, S.K. Nayar and P.N. Belhumeur,
SIGGRAPH 2005 Technical Sketch,
pp.00,
Aug, 2005.
[PDF] [bib] [©]

Image

Comparison of OLB and other Lighting Bases: Some reconstructed images (and their corresponding errors) for synthetic static objects which have been illuminated by Spherical Harmonic Lighting Basis (SHLB), Fourier Lighting Basis (FLB) and the proposed Optimal Lighting Basis (OLB). Note how the OLB performs significantly better.

Video

SIGGRAPH 2005 Sketch Video: This video shows how the optimal lighting basis is computed and several examples of video relighting. (With narration)

Slides

EGSR 2007 presentation     With videos (zip file)

Database

Time-Varying BRDF (TVBRDF) Database: We have acquired a variety of samples including watercolors, spray paints, oil paints, fabrics, cement, clay, plaster, joint compound dust, household dust and chocolate. A complete list of our 41 samples and the models used to fit their data can be found on the project page. Our goal is to use this database to first identify temporal trends in the estimated parameter values that are associated with each type of time-varying phenomenon (drying paint, drying wet surface, dust accumulation). Next, we propose analytic functions that model these temporal trends in parameter values. These models enable us to apply several of the above physical processes to novel materials.

Related Project

Multiplexed Illumination