Depth from Defocus

Structures of dynamic scenes can only be recovered using a real-time range sensor. Depth from defocus offers a direct solution to fast and dense range estimation. It is computationally efficient as it circumvents the correspondence problem faced by stereo and feature tracking in structure from motion. However, accurate depth estimation requires theoretical and practical solutions to a variety of problems including recovery of textureless surfaces, precise blur estimation and magnification variations caused by defocusing. In the first part of this project, both textured and textureless surfaces are recovered using an illumination pattern that is projected via the same optical path used to acquire images. The illumination pattern is optimized to ensure maximum accuracy and spatial resolution in computed depth. The relative blurring in two images is computed using a narrow-band linear operator that is designed by considering all the optical, sensing and computational elements of the depth from defocus system. Defocus invariant magnification is achieved by the use of an additional aperture in the imaging optics. A prototype focus range sensor has been developed that produces up to 512x480 depth images at 30 Hz with an accuracy better than 0.3%. Several experiments have been conducted to verify the performance of the sensor.

As a part of this project, we have also explored constant-magnification optics for imaging. Magnification variations due to changes in focus setting pose a problem for depth from defocus. The magnification of a conventional lens can be made invariant to defocus by simply adding an aperture at an analytically derived location. The resulting optical configuration is called "telecentric." We have shown that most commercially available lenses can be turned into telecentric ones. We have also conducted a detailed analysis of the photometric and geometric properties of telecentric lenses.

We have also addressed the problem of computing depth from defocus without the use of active illumination. We have developed a class of broadband operators that, when used together, provide invariance to scene texture and produce accurate and dense depth maps. Since the operators are broadband, a small number of them are sufficient for depth estimation of scenes with complex textural properties. In addition, a depth confidence measure is derived that can be computed from the outputs of the operators. This confidence measure permits further refinement of computed depth maps. Experiments have been conducted on both synthetic and real scenes to evaluate the performance of the proposed operators. The depth detection gain error is less than 1%, irrespective of the texture frequency.

Publications

"Rational Filters for Passive Depth from Defocus,"
M. Watanabe and S.K. Nayar,
International Journal on Computer Vision,
Vol.27, No.3, pp.203-225, May, 1998.
[PDF] [bib] [©]

"Telecentric Optics for Focus Analysis,"
M. Watanabe and S.K. Nayar,
IEEE Transactions on Pattern Analysis and Machine Intelligence,
Vol.19, No.12, pp.1360-1365, Dec, 1997.
[PDF] [bib] [©]

"Are Textureless Scenes Recoverable?,"
H. Sundaram and S.K. Nayar,
IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
pp.814-820, Jun, 1997.
[PDF] [bib] [©]

"Real-Time Focus Range Sensor,"
S.K. Nayar, M. Watanabe and M. Noguchi,
IEEE Transactions on Pattern Analysis and Machine Intelligence,
Vol.18, No.12, pp.1186-1198, Dec, 1996.
[PDF] [bib] [©]

"Minimal Operator Set for Passive Depth from Defocus,"
M. Watanabe and S.K. Nayar,
IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
pp.431-438, Jun, 1996.
[PDF] [bib] [©]

"Telecentric Optics for Computational Vision,"
M. Watanabe and S.K. Nayar,
DARPA Image Understanding Workshop (IUW),
pp.781-786, Feb, 1996.
[PDF] [bib] [©]

"Real-Time Computation of Depth from Defocus,"
M. Watanabe, S.K. Nayar and M. Noguchi,
Proceedings of The International Society for Optical Engineering (SPIE),
Vol.2599, pp.14-25, Jan, 1996.
[PDF] [bib] [©]

"Telecentric Optics for Constant Magnification Imaging,"
M. Watanabe and S.K. Nayar,
Technical Report, Department of Computer Science, Columbia University CUCS-026-95,
Sep, 1995.
[PDF] [bib] [©]

"Real-Time Focus Range Sensor,"
S.K. Nayar, M. Watanabe and M. Noguchi,
IEEE International Conference on Computer Vision (ICCV),
pp.995-1001, Jun, 1995.
[PDF] [bib] [©]

Videos

If you are having trouble viewing these videos in your browser, please save them to your computer first (by right-clicking and choosing "Save Target As..."), and then open them.

  Gestures:

This video shows the structure of a moving hand being computed in real time by the focus range sensor. The depth map is shown as a gray-coded image.

     
  Pouring Milk:

This video shows the structure of a cup and milk being poured out of it, computed in real time by the focus range sensor. The depth map is shown as a gray-coded image.

     
  Rolling Ball:

This video shows the structure of a rolling ball being computed in real time by the focus range sensor. A wire-frame representation of the depth map is also shown.

     
  Magnification Variation in Conventional Lens Due to Focusing:

This video shows how the magnification of a conventional lens varies as one changes the focus setting. The magnification variation is shown via the optical flow field that is generated when the focus setting is varied.

     
  Constant Magnification Imaging Using Telecentric Optics:

This video shows how the magnification of a telecentric lens remains unchanged (no optical flow) when the focus setting is varied.

     

Shape from Focus

Generalized Mosaicing