Generalized Mosaicing
Traditional image mosaicing uses a sequence of images of a scene captured by a moving camera. Generalized mosaicing is a simple and effective method for extracting additional information at each scene point. The key observation is that as a camera moves, it senses each scene point multiple times. We rigidly attach to the camera (either in front or behind the imaging lens) an optical filter with spatially varying properties. When this imaging system moves, each scene point is measured multiple times but under different optical settings. Fusing the data captured in the multiple images yields an image mosaic that includes additional information about the scene. This information can come in the form of extended dynamic range, high spectral quality, polarization, focus/distance sensing or enhancements to other dimensions of imaging.

Publications

"Generalized Mosaicing: Polarization Panorama,"
Y.Y. Schechner and S.K. Nayar,
IEEE Transactions on Pattern Analysis and Machine Intelligence,
Vol.27, No.4, pp.631-636, Apr, 2005.
[PDF] [bib] [©]

"Uncontrolled Modulation Imaging,"
Y.Y. Schechner and S.K. Nayar,
IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
Vol.II, pp.197-204, Jun, 2004.
[PDF] [bib] [©]

"Generalized Mosaicing: High Dynamic Range in a Wide Field of View,"
Y.Y. Schechner and S.K. Nayar,
International Journal on Computer Vision,
Vol.53, No.3, pp.245-267, Jul, 2003.
[PDF] [bib] [©]

"Generalized Mosaicing: Wide Field of View Multispectral Imaging,"
Y.Y. Schechner and S.K. Nayar,
IEEE Transactions on Pattern Analysis and Machine Intelligence,
Vol.24, No.10, pp.1334-1348, Oct, 2002.
[PDF] [bib] [©]

"Generalized Mosaicing,"
Y.Y. Schechner and S.K. Nayar,
IEEE International Conference on Computer Vision (ICCV),
Vol.1, pp.17-24, Jul, 2001.
[PDF] [bib] [©]

Images

  Generalized Mosaicing:

A spatially varying optical filter is rigidly attached to the camera. The filter can be attached either in front of or behind the imaging lens. As the camera moves, each scene point is seen through the filter under different optical settings. This allows one to reduce the redundancy that is inherent to traditional mosaicing and improve the resolution of the image along various imaging dimensions.

     
  High Dynamic Range Mosaic:

This picture shows the HDR mosaic computed using the captured 8-bit video sequence. While the original video has only 256 brightness levels, the computed mosaic has very high dynamic range as shown by the inset images.

     
  Linear Interference Filter:

This filter was attached to the camera to capture a multispectral mosaic.

     
  Multispectral Mosaic:

This picture shows the multispectral mosaic computed using the above video stream. The continuous spectral distribution of each point in the scene is computed. The multispectral mosaic can be used with a new illumination spectrum to compute a relit mosaic, as shown at the bottom.

     

Videos

If you are having trouble viewing these .mpg videos in your browser, please save them to your computer first (by right-clicking and choosing "Save Target As..."), and then open them.

  High Dynamic Range Mosaicing:

This video was captured by attaching a spatially varying neutral density filter to the camera and rotating the camera.

     
  Multispectral Mosaicing:

This video shows the raw 8-bit image sequence captured using the linear interference filter and a black and white camera.

     
  Polarization Mosaicing:

This video was captured using an optical filter with spatially varying polarization angle. The video can be used to compute the percentage and angle of polarization for each scene point. This information in turn can be used to detect materials and remove specularities (see publications for details).

     
  Focus Mosaicing:

This video was acquired by attaching a wedge-shaped optical block between the imaging lens and the image detector. The resulting imaging system has a spatially varying focal distance. As seen in the video, as the camera moves, each scene point varies in terms of its focus quality depending on its depth. The video is used to compute an all-focused mosaic and a depth map of the scene (see publications for details).

     

Slides

ICCV 2001 presentation     With videos (zip file)

High Dynamic Range Imaging: Multiple Exposures

Temporal Modulation Imaging

Radiometric Camera Calibration

Shape from Focus

Spherical Mosaics: Regular and Stereoscopic

Generalized Assorted Pixel Camera