The perception of transparent objects from images is known to be a very hard vision problem. Given a single image, it is difficult to even detect the presence of transparent objects in the scene. In this project, we explore what can be said about transparent objects by a moving observer. We show how features that are imaged through a transparent object behave differently from those that are rigidly attached to the scene. We present a novel model-based approach to recover the shapes and the poses of transparent objects from known motion. The objects can be complex in that they may be composed of multiple layers with different refractive indices. We have applied our algorithm to real scenes that include transparent objects and recovered the shapes and dimensions of the objects with high accuracy.
Refraction through Transparent Objects:
A ray of light interacts with a transparent object at two interfaces. The first interaction occurs when the ray passes from the surrounding medium having a refractive index of u1 into the transparent object having a refractive index u2. The second interaction occurs when the ray passes from the transparent object back into the surrounding medium. At each interface a portion of the ray is reflected and another portion is refracted.
The set of parallel incident rays from a distant feature are scattered by a transparent object in various directions. Some of these rays are captured by a moving camera, when they pass through the optical center of the camera. Given the intrinsic parameters and motion of the camera, we can compute the location (in 3D) of each captured ray in space. Note that if we knew the shape and pose of the object, these rays could be traced backwards to form a set of (outgoing) parallel rays. This holds true regardless of the complexity of the transparent object. It also remains valid for rays that go through total internal reflections (like the one imaged by the camera in the figure) as well as rays that do not pass through the transparent object (such as the bottom most one).
Transparent Cube and Moving Camera:
This video clip shows the nine views used to recover the shape of a transparent cube with rounded edges.
The initial guess shape overlaid on one of the images. The initial guess was a sphere (n=1.0).
An image of the actual cube used in the experiment shown to convey its shape. Its refractive index u is unknown.
The recovered shape of the cube (n=0.18).
This video clip shows the nine views used to recover the shape of an acrylic cylinder filled with water (4 interfaces).
The initial guess shape is overlaid on one of the images. The initial diameter of the cylinder was 200mm and the initial thickness of its wall was 20mm.
An image of the actual acrylic cylinder. It has a diameter of 117mm and wall thickness of 3mm.
The computed diameter of the cylinder is 116.1mm and the computed thickness of its wall is 2.3mm. The results are accurate even though this is a very complex case with 4 interfaces.