High Dynamic Range Imaging: Assorted Pixels
Multisampled imaging using assorted pixels is a general framework for using pixels on an image detector to simultaneously sample multiple dimensions of imaging (space, time, spectrum, brightness, polarization, etc.). The mosaic of red, green, and blue spectral filters found in most solid-state color cameras is one example of multisampled imaging. In this project, we show how multisampling can be used to explore other dimensions of imaging. Once such an image is captured, smooth reconstructions along the individual dimensions can be obtained using standard interpolation algorithms. Typically, this results in a substantial reduction of resolution (and, hence, image quality). One can extract significantly greater resolution in each dimension by noting that the light fields associated with real scenes have enormous redundancies within them, causing different dimensions to be highly correlated. Hence, multisampled images can be better interpolated using local structural models that are learned off-line from a diverse set of training images. The specific type of structural models we use are based on polynomial functions of measured image intensities. They are very effective as well as computationally efficient. We demonstrate the benefits of structural interpolation using three specific applications. These are traditional color imaging with a mosaic of color filters, high dynamic range monochrome imaging using a mosaic of exposure filters, and high dynamic range color imaging using a mosaic of overlapping color and exposure filters. Some of the images shown on this page courtesy of the Sony-Kihara Research Center, Inc.

Publications

print_paperentry_byid: more than 2 or no entry matches.


print_paperentry_byid: more than 2 or no entry matches.


print_paperentry_byid: more than 2 or no entry matches.


print_paperentry_byid: more than 2 or no entry matches.


print_paperentry_byid: more than 2 or no entry matches.


print_paperentry_byid: more than 2 or no entry matches.

Pictures

  Assorted Pixels:
This picture shows a few examples of multisampling using assorted pixels. (a) The popular Bayer color mosaic. (b) A mosaic of pixels with different transmittances. (c) A mosaic that includes different colors and transmittances. (d) A mosaic that includes different colors and linear polarizers with different orientations. (e) The change of image exposure over time. (f) The variation of color over space and exposure over space and time.
  Results for Spatially Varying Color:
This example compares interpolation results for the standard Bayer color mosaic obtained using standard interpolation and interpolation using structural models.
  Results for Spatially Varying Exposure:
This example shows how a high dynamic range image can be computed using a low dynamic range detector with spatially varying pixel exposures. The use of structural models for interpolation results in minimal loss of spatial resolution.
  Results of Spatially Varying Exposure and Color:
These examples show a high dynamic range color image computed from a low dynamic range image taken with spatially varying colors and exposures. Again, the structural models produce a higher quality interpolation.
  Prototype CCD:
This is the detector used in an assorted pixel prototype camera. The inset image shows the assortment of colors and exposures. (Image courtesy of Sony-Kihara Research Center, Inc.)
 

Images from the Prototype Camera:
This picture shows images of two scenes taken with a regular digital camera and the assorted pixel prototype camera. As can be seen, the assorted pixel camera produces images with greater scene details, less dark regions and less saturated regions. (Images courtesy of Sony-Kihara Research Center, Inc.)

Slides

ECCV 2002 presentation

Software

RASCAL: Radiometric Self-Calibration

Related Projects

Adaptive Dynamic Range Imaging

High Dynamic Range Imaging: Multiple Exposures

Radiometric Camera Calibration

Temporal Modulation Imaging

Generalized Mosaicing

Programmable Imaging: Micromirror Arrays