Super-Resolution: Jitter Camera

The resolution of videos can be computationally enhanced by moving the camera and applying super-resolution algorithms. However, a moving camera introduces motion blur, which limits the quality of super-resolution. We analyze this phenomenon and derive a theoretical result showing that motion blur severely degrades the performance of super-resolution. The conclusion is that in order to achieve any reasonable enhancement in resolution by using super-resolution, motion blur should be avoided.

To this end, we have developed a novel camera called the "jitter camera". The jitter camera produces shifts between consecutive video frames without introducing any motion blur. This is done by shifting the video detector instantaneously and timing the shifts to occur between pixel integration periods. The videos captured by the jitter camera are processed by an adaptive super-resolution algorithm that handles complex dynamic scenes in a robust manner. We show that the video resolution is significantly enhanced for stationary or slowly moving objects, while it is improved slightly or left unchanged for objects with fast and complex motions. The end result is a video that has a higher resolution than the captured one.

Publications

"Video Super-Resolution Using Controlled Subpixel Detector Shifts,"
M. Ben-Ezra, A. Zomet, and S.K. Nayar,
IEEE Transactions on Pattern Analysis and Machine Intelligence,
Vol.27, No.6, pp.977-987, Jun, 2005.
[PDF] [bib] [©]


"Jitter Camera: High Resolution Video from a Low Resolution Detector,"
M. Ben-Ezra, A. Zomet and S.K. Nayar,
IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
Vol.II, pp.135-142, Jun, 2004.
[PDF] [bib] [©]

Pictures

  Jitter Camera:
The jitter camera prototype has a lens, a board camera and computer-controlled micro-actuators. The actuators are synchronized with the integration cycle of the detector. This enables the actuators to shift the detector between consecutive integration periods and not during integration, thereby avoiding motion blur in the captured video.
  Resolution Chart Result:
By avoiding motion blur, the camera produces a significant increase in image resolution, as shown in this experiment in which a standard resolution target is used as the scene. Click on the thumbnail to see the images (produced by conventional and jitter cameras) in full resolution.
  Outdoor Scene Result:
Click on the thumbnail on the left to see the outdoor images (produced by conventional and jitter cameras) in full resolution. Note the boundaries of the figures on the pedestrian sign.
  Dynamic Scene Result:
Our adaptive super-resolution algorithm can robustly handle occlusions and multiple moving objects. Therefore, it can also increase the resolution of moving objects as shown by these images.

Videos

  CVPR 2004 Video:
This video compares raw video sequences captured using the jitter camera and the corresponding super-resolution videos. In particular, the video shows how our system handles complex dynamic scenes with multiple moving objects and occlusions.

Slides

CVPR 2004 presentation     With videos (zip file)

Related Projects

Motion Deblurring using Hybrid Imaging