Gigapixel
Computational Imaging[1]
|
|
 |
|
| Figure
1:
A 1.7 gigapixel image
captured using the implementation shown in Figure 4. The image
dimensions are 82,000 x 20,000 pixels, and the scene occupies a 126x32
degree FOV. From left to right, insets reveal the label of a resistor
on a PCB board, the stippling print pattern on a dollar bill, a
miniature 2D barcode pattern, and the fine ridges of a fingerprint on a
remote control. The insets are generated by applying a 60x-200x digital
zoom to the above gigapixel image. |
|
Today's high-resolution cameras capture images with pixel counts in the tens of millions. When digital cameras can produce images with billions of pixels, they will usher in a new era for photography. A gigapixel image has such a tremendous amount of information that one can explore minute details of the scene (see Figure 1). Gigapixel images are fascinating because they capture orders of magnitude more detail than the human eye, revealing information that was completely imperceptible to the photographer at the time of capture. At present, highly specialized gigapixel imaging systems are being developed for aerial surveillance [2]. Why are there no gigapixel cameras commercially available today? CMOS and CCD technologies have improved to the point that imaging sensors with pixels in the 1µm range have been demonstrated[3]. It is certainly within the reach of manufacturing technology to produce sensors with 1 billion pixels. On the other hand, it remains a huge challenge to design and manufacture lenses which have the resolving power to match the resolution of such a sensor. This is because the number of resolvable points for a lens, referred to as the Space-Bandwidth Product (SBP)[4], is fundamentally limited by geometrical aberrations. SBP is a unit-less quantity that tells us the number of distinct points which can be measured over a given FOV. Ideally, all lenses would be diffraction limited so that increasing the scale of a lens while keeping FOV fixed would increase SBP. Unfortunately, SBP reaches a limit due to geometrical aberrations. Scaling Laws for Lenses. Lohmann originally observed that lenses obey certain scaling laws that determine how resolution increases as a function of lens size[5]. Consider a lens with focal length f, aperture diameter D, and image size h by w. If we scale the lens by a factor of M, then f, D, h, and w are all scaled by M, but the F/# and FOV of the lens remain unchanged. If, when we scale the lens, the minimum resolvable spot size has not also increased by a factor of M, then we have increased the total number of points that can be resolved, thus increasing SBP. Figure 2 shows three different curves for SBP as a function of lens scale. The red Rd curve shows the ideal, diffraction limited case, where SBP increases quadratically with lens scale. Most lenses, however, are limited by geometrical aberrations. Then, as the green Rg curve shows, SBP eventaully reaches a plateau and stops increasing with lens scale. Since aberrations can be reduced by stopping down the lens aperture, lens designers typically increase F/# as a lens is scaled up. A general rule of thumb for conventional lens design is that F/# is made to increase like the cube root of lens scale. When this rule of thumb is applied, SBP no longer plateaus at the aberration limit, as shown in the blue Rf curve. We present a different approach to increase SBP - the use of computations to correct for geometrical aberrations. In conventional lens design, resolution is limited by the spot size of the lens. For a lens with aberrations, spot size increases linearly with the scale of the lens. For a computational imaging system, resolution is related to deblurring error. We observe, however, that for a lens with spherical aberrations, deblurring error does not increase linearly with lens scale. We use this remarkable fact to derive a scaling law that shows that computational imaging can be used to develop cameras with very high resolution while maintaining low complexity and small size. The magenta Rc curve in Figure 3 shows that, for a computational imaging system with a fixed SNR (i.e. fixed deblurring error), SBP scales more quickly with lens size than it does for conventional lens designs. |
|
 |
 |
| Figure
2.
A
plot
showing
how
Space-Bandwidth
Product
(SBP)
increases
as
a
function
of
lens
size
for
a perfectly diffraction limited lens (Rd),
a
lens with geometric aberrations (Rg), and a lens whose F/#
increases
with lens size (Rf). |
Figure
3.
A
new
scaling
law
for
computational
imaging
(Rc).
Note that Rc not
only improves upon the aberration limited curve Rg, it also
improves
upon the conventional lens design curve Rf without requiring F/# to increase
with lens scale. |
Proposed Architecture. By using a large ball lens, an array of planar sensors, and deconvolution as a post processing step, we are able to capture gigapixel images with a very compact camera. The key to our architecture lays in the size of the sensors relative to the ball lens. Together, a ball lens and spherical image plane produce a camera with perfectly radial symmetry. We approximate a spherical image plane with a tessellated regular polyhedron. A planar sensor is placed on each surface of the polyhedron. Relatively small sensors are used so that each sensor occupies a small FOV and the image plane closely approximates the spherical surface. As a result, our camera produces a PSF that is not completely spatially invariant, but comes within a close approximation. The first system we demonstrate consists solely of a ball lens and an array of planar sensors. We use a 100mm acrylic ball lens and a 5 megapixel 1/2.5" Lu575 sensor from Lumenera (see Figure 4). We emulate an image captured by multiple sensors by sequentially scanning the image plane using a pan/tilt motor. With this camera, a 1 gigapixel image can be generated over a roughly 60x40 degree FOV by tiling 14x14 sensors onto a 75x50mm image surface. When acquiring images with the pan/tilt unit, we allow a small overlap between adjacent images. Our first camera system is extremely compact, but it assumes there is no dead space between adjacent sensors. Sensors require at least some packaging around the active pixel area, which renders this solution impractical. A solution to overcome this problem is to introduce a secondary optic for each sensor, changing the system magnification so that the FOV of adjacent sensors overlaps slightly. This approach is taken in the implementation shown in Figure 5. |
|
 |
 |
| Figure
4.
Our
single
element
gigapixel
camera,
which
consists
solely
of
a
ball
lens
with
an
aperture
stop. A gigapixel image is captured by
sequentially translating a single 1/2.5", 5 megapixel sensor with a
pan/tilt motor. A final implementation would require a large array of
sensors with no dead space in between them. |
Figure
5.
A
multiscale
design
based
on
our
proposed
architecture.
An
array
of
relay
lenses
modifies
the
system magnification so that the FOV of
adjacent sensors overlaps slightly. The implementation is capable of
capturing a 15 megapixel region of a gigapixel image. A full gigapixel
camera requires 25x as many sensors and relay lenses. |
| A Single Element Design. The design
in Figure 4 is extremely compact, but impractical because adjacent
sensors must be packed without any dead space in between them. The
design in Figure 5 is practical enough for an implementation using
off-the-shelf components, but is much less compact. The size of this
system is limited by the package size of the sensor relative to the
active sensor area. Sensors with a package size that is only 1.5x
larger than the active sensor area are currently commercially
available. With these sensors, it is possible to build a gigapixel
camera that uses only a single optical element, as shown in the Zemax
raytrace of Figure 6. In this design, each sensor is coupled with a
smaller acrylic relay lens that decreases the focal length of the
larger acrylic ball lens. The relay lenses share a surface with the
ball lens, which means that it is possible to combine the entire
optical system into a single element that may be manufactured by
molding a single material, drastically simplifying the complexity (and
hence alignment) of the system. |
|
 |
 |
| Figure
6.
A
single
element
design
for
a
gigapixel
camera.
The
design
is
a
hybrid
between
the two implementations introduced in Figures 4 and 5.
Each sensor is coupled with a lens that decreases focal distance,
allowing FOV to overlap between adjacent sensors. |
|
|
Capturing the Complete Sphere. All of our proposed designs use a ball lens. A great advantage of using a ball lens is that, because it has perfect radial symmetry, a near hemispherical FOV can be captured. In fact, it can even be used to capture the complete sphere, as shown in Figure 7. This design is similar to the ones in Figures 4 and 5 with a large gap between adjacent lens/sensor pairs. Light passes through the gaps on one hemisphere, forming an image on a sensor located on the opposite hemisphere. As a result, the sensors cover the complete 2π FOV at the cost of losing roughly half the incident light. |
|
 |
|
| Figure 7. A design for a gigapixel camera with a 2π radian FOV. The design is similar to the implementation in Figures 4 and 5 with a large gap between adjacent lens/sensor pairs. Light passes through the gaps on one hemisphere, forming an image on a sensor located on the opposite hemisphere. | |
|
Captured Gigapixel Images. The following examples can be viewed in detail at gigapan.org. |
|
 |
Portrait [gigapan] Image Resolution: 65,000 x 25,000 = 1.6 gigapixels. |
 |
Still Life [gigapan] Image Resolution: 82,000 x 22,000 = 1.7 gigapixels. |
 |
New York Skyline [gigapan] Image Resolution: 110,000 x 12,000 = 1.4 gigapixels. |
References [1] O. Cossairt, D. Miau, S.K. Nayar. Gigapixel Computational Imaging, to appear in ICCP 2011 [2] Darpa at 50. "www.darpa.mil/Docs/1- 25013846_ Eprint_200811141152151.pdf", 2010. [3] K. Fife, A. El Gamal, and H. Wong. A 3MPixel Multi-Aperture Image Sensor with 0.7 µm Pixels in 0.11 µm CMOS. In IEEE ISSCC Conference, 2008. [4] J. Goodman. Introduction to Fourier optics. Roberts & Company Publishers, 2005. [5] A. W. Lohmann. Scaling laws for lens systems. Appl. Opt., 28(23):4996–4998, 1989. |
|
|
|
|