Natural illumination, realistic specular reflections, and complex cast
shadows are key features in determining the appearance of objects. However,
they have largely been ignored in computer vision algorithms. Indeed, most
algorithms assume Lambertian objects lit by a single point source, without
any explicit notion of visibility. Our goal is to derive the theoretical
foundations and mathematical equations and computational techniques to allow
much more robust computer vision algorithms that can work in outdoor lighting
conditions, with difficult specular objects, taking shadows into account.
| Primary Participants |
| Publications |
The theoretical results for the Lambertian case , and especially the 9 parameter model have been widely used to handle complex illumination in many areas of vision, including lighting-insensitive recognition, structure from motion, and photometric stereo. Recently, we have linked the convolution results with principal component analysis, deriving more compact subspaces and explaining some classic previous empirical results. Our results enable complex illumination on Lambertian matte surfaces to be handled with essentially the same complexity as point light sources. In more recent work, we have also developed a compact model of specularities, combining that with the Lambertian results to use specularities for the first time as a positive source of information in lighting-insensitive recognition. We have recently also looked at cast shadows, especially in natural 3D textures like moss, sponge or gravel, showing that in many canonical cases, a convolution result can be derived and used to explain some observations regarding lighting variability. A good overview of spherical harmonic lighting and the theoretical basis behind our methods is given in a recent book chapter. Most recently, we have derived a new class of frequency domain spherical harmonic identities . They can be used for direct transfer-based relighting, without formal inverse rendering, and to check the consistency of images to detect tampering or splicing.
 |
Analytic PCA Construction for Theoretical Analysis of Lighting Variability, Including Attached Shadows, in a Single Image of a Convex Lambertian Object PAMI Oct 2002,
pp 1322-1333.
We explain for the first time some classic empirical results on lighting variability, and take a first step toward analyzing many classic vision problems under complex lighting. Full Paper: PDF (.8M) |
|
|
Using Specularities for Recognition ICCV 03, pages 1512-1519 We present the first method for using specularities as a positive feature for lighting-insensitive recognition. The method is applied to very difficult objects like shiny crockery and wine glasses. Paper: PDF |
|
|
A Fourier Theory for Cast Shadows ECCV 04, pages I 146-162 ; PAMI Feb 05, pages 288-295 We show that cast shadows can be mathematically analyzed for many simple configurations, resulting in a standard convolution formula that can be derived analytically in 2D and analyzed numerically in 3D. The results help explain many effects of lighting variability in 3D textures and suggest new bases for that purpose. Paper: ECCV 04 , PAMI 05 |
 |
Modeling Illumination Variation with Spherical Harmonics
Book chapter in Face Processing: Advanced Modeling Methods (pages 385-424, 2005) The appearance of objects including human faces can vary dramatically with the lighting. We present results that use spherical harmonic illumination basis functions to understand this variation for face modeling and recognition, as well as a number of other applications in graphics and vision. Paper: PDF |
 |
A Theory of Spherical Harmonic Identities for BRDF/Lighting
Transfer and Image Consistency ECCV 06 vol IV, pp 41-55, PAMI 07. We develop new mathematical results based on the spherical harmonic convolution framework for reflection, deriving novel identities, which are the angular frequency domain analogs to common spatial domain invariants such as reflectance ratios. Paper: PDF |