Coded Aperture Pairs for Depth from Defocus

The classical approach to Depth From Defocus (DFD) uses lenses with circular apertures for image capture. We show in this project that the use of a circular aperture severely restricts the accuracy of DFD. We derive a criterion for evaluating a pair of apertures with respect to the precision of depth recovery. This criterion is optimized using genetic algorithms and gradient descent search to arrive at a pair of high resolution apertures. The two coded apertures are found to complement each other in the scene frequencies they preserve. This property enables them to not only recover depth with greater fidelity but also obtain a high quality all-focused image from the two captured images. Extensive simulations as well as experiments on a variety of scenes demonstrate the benefits of using the coded apertures over conventional circular apertures.

This research was funded in part by ONR awards N00014-08-1-0638 and ONR N00014-08-1-0329.

Publications

"Coded Aperture Pairs for Depth from Defocus,"
C. Zhou, S. Lin, and S. K. Nayar,
IEEE International Conference on Computer Vision (ICCV),
Oct, 2009.
[PDF] [bib] [©]

Images

  Issues with single aperture solutions:

A variety of aperture patterns have been introduced for different applications since the early 1960s. For defocus deblurring, people want patterns to have less zero-crossing frequencies in the Fourier domain. On the other hand, people want patterns to have more zero-crossing frequencies for depth estimation. A basic limitation of using a single coded aperture is that aperture patterns with a broadband frequency response are needed for optimal defocus blurring but are less effective for depth estimation, while patterns with zero-crossings in the Fourier domain yield better depth estimation but exhibit a loss of information for deblurring.

     
  Formulating DFD as an optimization problem:

Given two (or more) captured images with different defocus characteristics, the objective of DFD is to find the depth d and the focused image f_0 that minimize the energy function E. To make the estimation less sensitive to image noise and more robust to weak texture, the curve E(d) should be as steep as possible.

     
  Evaluation criterion of coded aperture pairs for DFD:

The use of coded aperture pairs changes the steepness of the energy curves. Since steeper E curves imply more reliable depth estimations, we define the criterion for aperture pair evaluation as the steepness of the corresponding energy curves.

     
  Optimized circular aperture pairs:

A typical DFD method captures two images, one with a large circular aperture and one with a small circular aperture. We first use our derived evaluation criterion to determine the optimal radius ratio of circular aperture pairs. For pillbox aperture patterns, we find the best ratio to be 1.5 . For Gaussian aperture patterns, the best ratio is 1.70.

     
  Optimized Coded Aperture Pairs:

Pattern optimization is known to be a challenging problem. In this paper, we propose a two-step optimization strategy. First, we employ the genetic algorithm proposed in [Zhou and Nayar, ICCP 2009] to find the optimized binary aperture at a low resolution of 11 11. Then, we scale up the 11 11 solution to 13 13 and then refine the solution using gradient descent optimization. This scale-and-refine process is repeated until reaching a resolution of 3333.

     
  Analysis of the optimized coded aperture pair:

Compared to a conventional circular aperture pair, the optimized coded aperture pair exhibits an E curve with a more pronounced minimum, which leads to depth estimation that has lower sensitivity to image noise and greater robustness to scenes with subtle texture. In addition, our optimized apertures are found to have complementary power spectra in the frequency domain, with zero-crossings located at different frequencies for each of the two apertures. Owing to this property, the two apertures thus jointly provide broadband coverage of the frequency domain. This enables us to also compute a high quality all-focused image from the two captured defocused images.

     
  Implementation:

We printed our optimized pair of aperture patterns on high resolution (1 micron) photomasks, and inserted them into two Canon EF 50mm f/1.8 lenses. These two lenses are mounted to a Canon EOS 20D camera in sequence to take a pair of images of each scene.

     
  Experimental results - Campus view:

We compare the optimized coded aperture pair with the conventional large/small circular apertures pair. Note that we achieve the much higher quality depth map and all-focused image by simply changing the aperture patterns.

     
  Experimental results - Desktop:

Two images of the desktop are captured by using the optimized coded aperture pair for the depth map and all-focused image recovery.

     
  Experimental results - Inside a book store:

Two images are captured inside a book store by using the optimized coded aperture pair for the depth map and all-focused image recovery. The ground truth image is captured by using a small aperture f/16 and a long exposure time.

     
  Supplementary Material (PDF):

The supplementary material PDF contains additional, high-quality results.

     

Slides

ICCV 2009 presentation     With videos (zip file)

Depth from Defocus

Flexible Depth of Field Photography

What are Good Apertures for Defocus Deblurring?

Depth from Diffusion