Relighting Framework Sebastian Enrique [ senrique@cs.columbia.edu ], Ravi Ramamoorthi [ ravir@cs.columbia.edu ] |
Introduction |
Relighting Framework is a framework for synthetic and real scenes image-based relighting with a fixed viewpoint but with any point / area lights and environment lighting conditions. The method consists of a large preprocessing stage, in which sets of images with different lighting conditions are provided, and two passes of PCA (Principal Component Analysis) compress data and generate the final matrices for each color component. Those matrices are then used in realtime to calcuate the correct image appeareance given the novel lighting conditions. The Relighting Framework is based on the Lighting Sensitive Display by Shree Nayar, Peter Belhumeur, and Terry Boult, and extends it to use environment maps. |
Image Adquisition |
Images can be synthetic or real photographs. The viewpoint should be fixed. The target object should be in the center of an imaginary cube. Each face of the cube should be divided in a grid of f x f elements. So, those 6 x f x f positions should be used to place one distant point light and take -render- 6 x f x f pictures -images-, i.e., one image for each light position. |
Preprocessing Stage |
First PCA Where L^{j} is an b x n matrix which we call the block lighting coefficients. Let the lighting coefficient matrix L be formed by stacking all of the m L^{j} block matrices. Likewise, let the collection of image bases for all blocks be denoted by E. The collection of submatrices within E and L contain all the information needed to approximate the collection of images corresponding to the n lighting directions. Second PCA Where U is an (m x b) x q column-orthogonal matrix which we call the lighting coefficient bases, V is a q x n matrix which we call the compressed coefficient matrix and q denotes the number of linear bases kept to approximate L. |
Realtime Stage |
We represent the illumination as the s vector field, of dimension n x 1. Each element in s is the contribution of the corresponding original distant point light image. To render an image of the scene as if it were illuminated by s we appeal to the previously calculated matrices E, U, and V. First, we compute a compressed coefficient vector as the product V s. Next, we compute a lighting coefficient vector as the product U V s. Note that the vector U V s has dimensions (m x b) x 1 and that the subvector l^{j} given by the rows b x (j - 1) + 1 through b x j of U V s contain the coefficients necessary for rendering the jth block. Thus, the subvectors can be unstacked and used to render the jth block of the displayed images as the product E^{j} l^{j}. This process is repeated for all m blocks. Point lights are mapped directly to one element of the s vector. Area lights are mapped to more than one element of the s vector. If the area light with intensity k occupies e elements, the contribution on each element is k / e. High-Dynamic Range Cube Map texels are interpolated to avoid problems when the resolution of each face of the cubemap is different from the quantity of images taken from each face of the object. n texels are taken according to the corresponding directions and mapped onto s. Texels contributions are weighted by the solid angle. |
Download |
Preprocess Stage [ preprocess.zip ] [ readme2.txt ] [ relighting.m ] Includes Matlab source function for preprocessing and output files and input images for Nicole (both shadowed and unshadowed) and David. Compressed in zip format at ~260MB. |
RCafe Presentation [ relightingv02.ppt ] Powerpoint presentation used in November 2003 in a RCafe talk to show the work in progress at that moment. ~3MB. |
Realtime Binary [ binary.zip ] [ readme.txt ] Includes executable file for realtime relighting, 10 HDR environment maps, data sets for Nicole (both shadowed and unshadowed) and David, and batch files to run every possible combination with included datasets. Compressed in zip format at ~240MB. |
Source Code [ sourcecode.zip ] Includes Visual C++ 6.0 project for realtime relighting and Matlab source function for preprocessing stage. Compressed in zip format at ~50KB. |
References |
Lighting Sensitive Display [ pdf ] Nayar, S., Belhumeur, P., Boult, T. 2001. In Technical Sketch, Siggraph 2001, page 218. |
Paul Debevec [ http://www.debevec.org/ ] Light probes (HDR Cubic Environment Maps) were taken from Paul Debevec's website. |
^{1} : Restricted site. |