What are Good Apertures for Defocus Deblurring?

Defocus deblurring is an old and important problem in computer vision. It is well known that the quality of a deblurred image is closely related to the point spread function, which is determined by the aperture pattern. For example, a conventional circular aperture results in a point spread function that severely attenuates high frequencies and has many zero-crossings in the frequency domain, so that some frequencies simply cannot be recovered from the defocused image and noise in the deblurred image is greatly exaggerated. The design of aperture patterns has been studied for decades in several fields, including, optics, astronomy, computer vision, and computer graphics. However, previous attempts at designing apertures have been based on simple and intuitive criteria, rather than on the optimality of the results.

In this project, we develop a comprehensive framework for evaluating and optimizing aperture patterns for defocus deblurring. Our criterion explicitly accounts for the effects of image noise and the statistics of natural images. This criterion has been used to assess a set of existing patterns in terms of defocus deblurring. Moreover, based on our criterion, we have developed a genetic algorithm that converges very quickly to near-optimal patterns of coded aperture. Extensive simulations and experiments show that our optimized coded apertures significantly outperform previously proposed ones.

Publications

"What are Good Apertures for Defocus Deblurring?,"
C. Zhou and S. K. Nayar,
IEEE International Conference on Computational Photography,
Apr, 2009.
[PDF] [bib] [©]

Images

  Formulation of Defocus:

The formation of an out-of-focus image (f) can be written as a convolution of the focused image (f_0) by the point spread function (k) plus image noise (xi). It is well known that a conventional circular aperture results in a point spread function that severely attenuates high frequencies and has many zero-crossings in the frequency domain. As a result some frequencies simply cannot be recovered by deconvolution and noise in the deblurred image is greatly exaggerated by the process of deconvolution.

     
  The power spectra of different patterns:

There are many possible aperture designs, each giving a different profile in the frequency domain. If the point spread function has low response at a certain frequency then the recorded image will have a lower signal-to-noise ratio, and the deblurred image will as well. Because of this, one may want a pattern which has high responses everywhere in the frequency domain. Unfortunately, the frequency response of a point spread function must have finite overall energy, and therefore cannot be high everywhere. Then, how should we measure the pattern optimality?

     
  Our methodology:

In this project, we will not assess aperture patterns directly based on the shape of the power spectra. Instead, we are going to propose an evaluation criterion R(K) which reflects the expected quality of the recovered image when the pattern (K) is used. To be more specific, in our framework, the input image (F_0) and image noise (eta) are regarded as two random vectors, and R(K) is defined as the expectation of the distance between the ground truth focused image and the recovered image.

     
  Our assumptions:

Note that the expectation of deblurring quality is related to the distribution of the latent focused image, the noise model, the deblurring algorithm, and the measurement of image quality. To make the problem more tractable, we assume a distribution of natural images, Gaussian white noise, optimal linear deblurring algorithm, and use L2 distance as a quality measurement.

     
  Evaluation Criterion of Aperture Patterns:

A concise and comprehensive evaluation criterion of aperture patterns has been derived. Given the Fourier transform of any PSF, K, the proposed function R(K) indicates the expectation of its deblurring performance. The lower the R value is, the better the deblurring performance. Two observations can be made by analyzing this formulation. One, the optimality of a pattern is highly related to the noise level. Two, different frequencies are not equally important for deblurring.

     
  Performance Comparison Using the Proposed Criterion:

The proposed criterion was used to evaluate many existing aperture patterns at various noise levels. Our evaluation shows that the conventional circular aperture tends to be superior when image noise level is high, while coded apertures (such as our "Image pattern" and Veeraraghavan et al.'s pattern) can do much better when image noise level is low.

     
  Aperture Optimization via Genetic Algorithm:

Even with our concise evaluation criterion, finding the optimal aperture pattern remains a challenging problem. While the aperture pattern is evaluated in the frequency domain, it must satisfy several physical constraints in the spatial domain. We therefore resort to a numerical search approach and optimize the pattern using a genetic algorithm. Experiments show that for patterns of resolution 13 x 13 or less, the genetic algorithm can converge quickly.

     
  Convergence Analysis of the Genetic Algorithm:

The proposed genetic algorithm is compared with a randomized linear search, commonly used in previous work. Each algorithm is repeated 10 times. We see that our genetic algorithm converges quickly to a low value for aperture criterion metric. Also, the results of different runs of the genetic algorithm are quite similar, indicating that they may all lie close to the optimum aperture.

     
  The Optimized Patterns:

We present the optimized patterns for eight different noise levels and their quality measured by our criteria. It is interesting to note that the optimized aperture patterns get more structured with increase in noise.

     
  The Implementation:

We printed our optimized aperture patterns as well as several other patterns in high resolution (1 micron) on a photomask sheet. To experiment with a specific aperture pattern, we inserted it into a Canon EF 50mm f=1.8 lens.

     
  Comparison Results using CZP Resolution Chart:

We placed a CZP resolution chart at the distance of 150cm from the lens, and captured images using the five different apertures: circular pattern, our "image pattern", the pattern from Levin et al., the pattern from Veeraraghavan et al., and one of our optimized patterns. The same exposure time was used for all the acquisitions. Note that our optimized pattern gives the sharpest deblurred image with least artifacts and image noise.

     
  Results for the Shrek Image:

On the left is an out-of-focus Shrek image captured using the optimized aperture pattern, while on the right is the deblurring result.

     
  Results for the Street Scene:

On the left is an out-of-focus street scene captured using the optimized aperture pattern, while on the right is the deblurring result. Although some ringing artifacts can be seen in our deblurred images, significant details are recovered by the use of our optimized pattern. Note that in the captured image, the degree of defocus is so great that many scene details are virtually invisible.

     

Flexible Depth of Field Photography

Depth from Defocus