Active Refocusing of Images and Videos

In this project, we present a system for refocusing images and videos of dynamic scenes using a novel, single-view depth estimation method. Our method for obtaining depth is based on the defocus of a sparse set of dots projected onto the scene. In contrast to other active illumination techniques, the projected pattern of dots can be removed from each captured image and its brightness easily controlled in order to avoid under- or over-exposure. The depths corresponding to the projected dots and a color segmentation of the image are used to compute an approximate depth map of the scene with clean region boundaries. The depth map is used to refocus the acquired image after the dots are removed, simulating realistic depth of field effects. Experiments on a wide variety of scenes, including close-ups and live action, demonstrate the effectiveness of our method.

Publications

"Active Refocusing of Images and Videos,"
F. Moreno-Noguer, P.N. Belhumeur and S.K. Nayar,
ACM Trans. on Graphics (also Proc. of ACM SIGGRAPH),
Aug, 2007.
[PDF] [bib] [©]

Images

  Active Refocusing Steps:

(a) Image acquired by projecting a sparse set of illumination dots on the scene. (b) The dots are automatically removed. (c) The defocus of the dots is used to compute a sparse depth map. (d-f) Color segmentation of the image is used to compute a complete depth map of the scene with sharp boundaries. (g-i) The depth map and the dot-removed image are used to smoothly refocus the scene. (j) The refocusing can also be done for an image taken immediately before or after, with new lighting.

     
  Controlling the Brightness of the Projected Dots:

Top: Images of a square dot projected onto different depths. Bottom: The radiance Iw of a projected dot may be controlled by changing the parameters of the setup, such as the distance uf of the focal plane, or the with w of the projected dots. By doing this, we can avoid under- or over-exposure, and ensure that the brightness of all the points in the image are within the dynamic range of the camera.

     
  Realistic Refocusing:

In order to render realistic depth of field effects we propose an algorithm which considers partial occlusions (for object boundaries, different parts of the lens may “see” different views). Left: original all-focused image. Right: refocusing results obtained with the proposed algorithm, where the virtual focal plane is placed on the background of the scene. Observe that the results are highly realistic, especially at the depth discontinuities.

     

Video

If you are having trouble viewing these .mov videos in your browser, please save them to your computer first (by right-clicking and choosing "Save Target As..."), and then open them.

  SIGGRAPH 2007 Video:

This video shows all our examples of image and video relighting based on the defocus of a sparse set of dots projected onto the scene. (With narration)

     

Projection Defocus Analysis

Depth from Defocus

Shape from Focus

Projection Defocus Analysis