Focal Sweep Videography with Deformable Optics

A number of cameras have been introduced that sweep the focal plane using mechanical motion. However, mechanical motion makes video capture impractical and is unsuitable for long focal length cameras. In this project, we present a focal sweep telephoto camera that uses a variable focus lens to sweep the focal plane. Our camera requires no mechanical motion and is capable of sweeping the focal plane periodically at high speeds. We use our prototype camera to capture EDOF videos at 20fps, and demonstrate space-time refocusing for scenes with a wide depth range. In addition, we capture periodic focal stacks, and show how they can be used for several interesting applications such as video refocusing and trajectory estimation of moving objects.

This research was supported in parts by ONR MURI Award No. N00014-08- 1-0638 and ONR Award No. N00014-11-1-0285.

Publications

"Focal Sweep Videography with Deformable Optics,"
D. Miau, O. Cossairt and S.K. Nayar,
IEEE International Conference on Computational Photography (ICCP),
pp.1-8, Apr, 2013.
[PDF] [bib] [©]

  Gallery of Long-Distance Space-Time Refocusing Results:

This gallery includes four interactive photos captured by our telephoto focal sweep camera. The scene distances of these photos range from 180m to more than 800m away from the camera. When refocusing a photo, a user can observe the small motion happened within the duration of focal sweep.

     

Images

  Focusing for a Deformable Lens:

This figure shows rays traced through the same deformable lens with different focal lengths. (a) and (b) show rays traced through the same deformable lens with different focal lengths. (a) A scene point s, at a distance o from the lens, is imaged in perfect focus at location m on a sensor at a distance i from the lens. (b) If the focal length of the lens is changed such that an image in perfect focus is at a distance i' from the lens, s is imaged as a blurred circle with diameter b centered around m.

     
  Focal Sweep with a Moving Sensor versus a Deformable Lens:

This figure shows two implementations of a telephoto focal sweep system using a 800mm lens. On the top is an implementation with a moving sensor. Note the complexity and the size of the mechanism required to translate the sensor at high speeds. The long sensor travel poses significant engineering challenges, such as susceptibility to vibrations. On the bottom is a compact implementation with a deformable lens, which is capable of focusing at high speeds and operating at high frequencies. In this project, we show several interesting applications enabled by focal sweep with deformable optics.

     
  Prototype Telephoto Focal Sweep Camera:

Our prototype camera consists of a Canon EF 800 f/5.6L lens, an Optotune EL-10-30 tunable lens, and a Point Grey Flea 3 image sensor. The Optotune tunable lens is attached to the mounting flange of the Canon telephoto lens.

     
  Extended Depth of Field Experimental Result of a Static Scene:

This figure compares fixed focus images with a focal sweep image captured using our prototype camera. In the scene, the left and the right resolution charts are placed at the distances of 65m and 75m away from the camera, respectively. For the fixed focus images, the depth of field is so narrow that one of the two charts is blurred beyond recognition. The image in the bottom is a focal sweep image after deblurring. The insets corresponding to the focal sweep image show that details from both resolution charts are preserved in the EDOF image.

     
  Subsampling a Periodic Focal Stack for Video Refocusing:

In this example the periodic focal stack includes two people walking toward the camera at different speeds. (a, b, c) show three frames of a generated video in which the person on the right is kept in focus. (d, e, f) show three frames of a generated video in which the person on the left is kept in focus. (g) shows the estimated depths of the focal plane (green line) and the two people (blue and red lines) over time. By selecting the frames captured at the times when the focal plane coincides with the depth of a person, one can generate a video in which focus follows the person. The colored markers indicate the locations corresponding to the frames shown above.

     

Videos

If you are having trouble viewing these .mp4 videos in your browser, please save them to your computer first (by right-clicking and choosing "Save Target As..."), and then open them.

  Comparison between a Fixed Focus Video and a Focal Sweep Video:

The two videos captured a person walking from 75m to 65m away from the camera. The video on the left is a conventional video focused at 70m, and the video on the right is an EDOF video captured with focal sweep. Note that in the conventional video the person's face was only focused in the middle of the depth range, due to the narrow depth of field associated with the telephoto lens. In contrast, in the EDOF video the person's face stayed focused throughout the entire depth range. The numbers in the upper right corner of each video indicate the distances of the person.

     
  Video Refocused to Follow the Man:

This refocused video, generated from the periodic focal stack discussed in the paper, has its focus following the man on the left. The plot in the bottom of the video indicates the estimated depths of the focal plane and the two people in the scene over time. Note that the depth of the man decreases rapidly as the video progresses. As the distance between the two people increases, only the man's face remains in focus.

     
  Video Refocused to Follow the Woman:

This refocused video, generated from the same periodic focal stack discussed in the paper, has its focus following the woman on the right. Note that the man's face gradually becomes out of focus as the video progresses, while the woman's face always stays in focus.

     

Slides

ICCP 2013 presentation

Flexible Depth of Field Photography

Focal Sweep Camera (Fosera)