Generalized Assorted Pixel Camera

We propose the concept of a generalized assorted pixel (GAP) camera, which enables the user to capture a single image of a scene and, after the fact, control the trade-off between spatial resolution, dynamic range and spectral detail. The GAP camera uses a complex array (or mosaic) of color filters.

The following are the main contributions of our work. (a) We develop an optimization method to arrive at the spatial and spectral layout of the color filter array of a GAP camera. The cost function that we optimize includes terms related to colorimetric/spectral reproduction, dynamic range, and signal-to-noise ratio (SNR). (b) We develop a novel anti-aliasing algorithm for reconstructing the under-sampled channels of the image with minimal aliasing. Our approach is to use a sub-micron pixel size to avoid aliasing for some of the channels. The high frequency content from these channels are then used to remove aliasing from the remaining (under-sampled) channels. (c) We have developed software that enables a user to capture a single image and then control the trade-off of spatial resolution to generate a variety of images. The output image can be monochrome, HDR monochrome, RGB, HDR RGB, or multispectral. (d) We have used high quality multispectral data as our ground truth as well as to verify our optimized mosaic and reconstruction algorithm.

This is a joint project with Sony Corporation.

A database of multispectral images is available here.

Publications

"Generalized Assorted Pixel Camera: Post-Capture Control of Resolution, Dynamic Range and Spectrum,"
F. Yasuma, T. Mitsunaga, D. Iso, and S.K. Nayar,
Technical Report, Department of Computer Science, Columbia University CUCS-061-08,
Nov, 2008.
[PDF] [bib] [©]

Interactive GAP Demos

  Flowers Scene:

An interactive demo showing different outputs that can be generated from the "flowers" GAP mosaic (simulated using a multispectral image).

Requirements: Adobe Flash Player.

     
  Stuffed Toy and Chart Scene:

An interactive demo showing different outputs that can be generated from the "stuffed toy and chart" GAP mosaic (simulated using a multispectral image).

Requirements: Adobe Flash Player.

     

Images

  MTF of Submicron Pixel Image Sensor:

We simulate the optical maximum resolution limit determined by the diffraction of an ideal (aberration-corrected) lens. The resolution performance of a sensor with submicron pixels exceeds the optical resolution limit.

     
  GAP Mosaic:

The pixels marked 'a', 'b', and 'c' do not suffer from aliasing because the optical resolution limit is 0.25fs due to diffraction. These aliasing-free pixels are used for reconstruction of high resolution images, and for removing the aliasing artifacts of other under-sampled pixels.

     
  The Spectral Responses of the Seven Optimized Filters:

We develop an optimization method to arrive at the spatial and spectral layout of the color filter array of a GAP camera. The cost function that we optimize includes terms related to colorimetric and spectral reproduction, dynamic range, and signal-to-noise ratio (SNR).

     
  Comparison with Other Mosaics:

Each of the previously proposed mosaics have been used to generate one specific type of output image. In contrast, our goal is to create a mosaic that lends itself to post-capture control over the output image.

     
  Anti-Aliasing Results:

The results of our anti-aliasing algorithm (simulated with N = f/5.6, p=1.0um). Left: Low exposure RGB image computed from the secondary filters without anti-aliasing (false color artifacts caused by aliasing are observed). Right: Low exposure RGB image obtained after anti-aliasing

     
  Result for Flowers Scene:

The texture and color of saturated regions in the monochrome and RGB images become visible in the corresponding HDR images. As expected, one can see more detail in the HDR monochrome images than in the HDR RGB images.

     
  Result for Stuffed Toy and Chart Scene:

Our algorithm has some limitations that we plan to investigate in future work. For instance, when the primary pixels are saturated, anti-aliasing cannot be applied to the secondary pixels and the spectral reflectance distribution cannot be fully reconstructed in the saturated regions.

     
  Result for Skin Detection:

The scene shown is composed of a real face (i.e., with real skin) on the right and a photo of the same face (i.e., on printed paper) on the left. These two faces (real and fake) are difficult to distinguish using the RGB image -- skin detection based on color analysis finds both faces even though only one of them is real. In contrast, skin detection applied to the multispectral image computed from the GAP image results in the desired result -- only the real face is found, as pixels within it have the spectrum of real skin.

     

Database

  Multispectral Image Database:

We present a database of multispectral images that were used to emulate the GAP camera. This database spans a wide variety of real world materials.

     

High Dynamic Range Imaging: Multiple Exposures

High Dynamic Range Imaging: Assorted Pixels

Multispectral Imaging Using Multiplexed Illumination

Generalized Mosaicing