What are Good Apertures for Defocus Deblurring? 

Defocus deblurring is an old and important problem in computer vision.
It is well known that the quality of a deblurred image is closely related to the
point spread function, which is determined by the aperture pattern. For example, a conventional circular aperture
results in a point spread function that severely attenuates high frequencies and has many zerocrossings
in the frequency domain, so that some frequencies simply cannot be recovered from the defocused image
and noise in the deblurred image is greatly exaggerated.
The design of aperture patterns has been studied for decades in several fields,
including, optics, astronomy, computer vision, and computer graphics. However,
previous attempts at designing apertures have been based on simple and
intuitive criteria, rather than on the optimality of the results.
In this project, we develop a comprehensive framework for evaluating and
optimizing aperture patterns for defocus deblurring. Our criterion explicitly
accounts for the effects of image noise and the statistics of natural images.
This criterion has been used to assess a set of existing patterns in terms of defocus deblurring.
Moreover, based on our criterion, we have developed a genetic algorithm that converges
very quickly to nearoptimal patterns of coded aperture. Extensive simulations and experiments show that
our optimized coded apertures significantly outperform previously proposed ones. 
Publications
"What are Good Apertures for Defocus Deblurring?," C. Zhou and S. K. Nayar, IEEE International Conference on Computational Photography, Apr. 2009. [PDF] [bib] [©] [Project Page]

Images


Formulation of Defocus:
The formation of an outoffocus image (f) can be written as a
convolution of the focused image (f_0) by the point spread function (k) plus
image noise (xi). It is well known that a conventional circular aperture
results in a point spread function that severely attenuates high
frequencies and has many zerocrossings in the frequency domain. As a
result some frequencies simply cannot be recovered by deconvolution
and noise in the deblurred image is greatly exaggerated by the process
of deconvolution.






The power spectra of different patterns:
There are many possible aperture designs, each giving a different
profile in the frequency domain. If the point spread function has low
response at a certain frequency then the recorded image will have a
lower signaltonoise ratio, and the deblurred image will as well.
Because of this, one may want a pattern which has high responses everywhere in the frequency domain.
Unfortunately, the frequency response of a point spread function must
have finite overall energy, and therefore cannot be high everywhere. Then, how should
we measure the pattern optimality?






Our methodology:
In this project, we will not assess aperture patterns directly based on the shape of the power spectra.
Instead, we are going to propose an evaluation criterion R(K) which reflects the expected quality of the recovered image when the pattern (K) is used.
To be more specific, in our framework, the input image (F_0) and image noise (eta) are regarded as two random vectors, and
R(K) is defined as the expectation of the distance between the ground truth focused image and the recovered image.






Our assumptions:
Note that the expectation of deblurring quality is related to the distribution of the latent focused image, the noise model, the deblurring
algorithm, and the measurement of image quality. To make the problem
more tractable, we assume a distribution of natural images, Gaussian white noise, optimal
linear deblurring algorithm, and use L2 distance as a quality measurement.






Evaluation Criterion of Aperture Patterns:
A concise and comprehensive evaluation criterion of aperture patterns
has been derived. Given the Fourier transform of any PSF, K, the
proposed function R(K) indicates the expectation of its deblurring
performance. The lower the R value is, the better the deblurring
performance. Two observations can be made by analyzing this
formulation. One, the optimality of a pattern is highly related to the
noise level. Two, different frequencies are not equally important for
deblurring.






Performance Comparison Using the Proposed Criterion:
The proposed criterion was used to evaluate many existing aperture patterns at
various noise levels. Our evaluation shows that the conventional
circular aperture tends to be superior when image noise level is high, while
coded apertures (such as our "Image pattern" and Veeraraghavan et al.'s
pattern) can do much better when image noise level is low.






Aperture Optimization via Genetic Algorithm:
Even with our concise evaluation criterion, finding the optimal aperture
pattern remains a challenging problem. While the aperture pattern is evaluated
in the frequency domain, it must satisfy several physical constraints in the spatial
domain. We therefore resort to a numerical search approach and optimize the
pattern using a genetic algorithm. Experiments show that for patterns of
resolution 13 x 13 or less, the genetic algorithm can converge quickly.






Convergence Analysis of the Genetic Algorithm:
The proposed genetic algorithm is compared with a randomized linear search,
commonly used in previous work. Each algorithm is repeated 10 times. We see
that our genetic algorithm converges quickly to a low value for aperture
criterion metric. Also, the results of different runs of the genetic algorithm
are quite similar, indicating that they may all lie close to the optimum
aperture.






The Optimized Patterns:
We present the optimized patterns for eight different noise levels and their
quality measured by our criteria. It is interesting to note that the optimized
aperture patterns get more structured with increase in noise.






The Implementation:
We printed our optimized aperture patterns as well as several other patterns
in high resolution (1 micron) on a photomask sheet. To experiment with a
specific aperture pattern, we inserted it into a Canon EF 50mm f=1.8 lens.






Comparison Results using CZP Resolution Chart:
We placed a CZP resolution chart at the distance of 150cm from the lens, and
captured images using the five different apertures: circular pattern, our
"image pattern", the pattern from Levin et al., the pattern from Veeraraghavan
et al., and one of our optimized patterns. The same exposure time was used for
all the acquisitions. Note that our optimized pattern gives the sharpest
deblurred image with least artifacts and image noise.






Results for the Shrek Image:
On the left is an outoffocus Shrek image captured using the optimized aperture pattern, while on the right is the deblurring result.






Results for the Street Scene:
On the left is an outoffocus street scene captured using the optimized aperture pattern,
while on the right is the deblurring result. Although some ringing artifacts can be seen in our deblurred images,
significant details are recovered by the use of our optimized pattern. Note
that in the captured image, the degree of defocus is so great that many scene
details are virtually invisible.





Flexible Depth of Field Photography
Depth from Defocus


