Time Varying Surface Appearance

In this project, we are interested in acquiring, modeling and rendering time-varying surface appearance. Traditional computer graphics and computer vision algorithms generally assume that the appearances of surfaces remain constant over time. However, there are a number of natural processes that cause surface appearance to vary dramatically, such as the burning of wood, the wetting and drying of rocks and fabrics, the decay of fruit skins, and the corrosion and rusting of steel and copper. We have captured the first time-varying surface appearance database (with 26 samples), which includes a variety of natural processes -- burning, drying, decay and corrosion. We have developed a novel Space-Time Appearance Factorization (STAF) model, which factors space- and time-varying appearance effects. The STAF model includes an overall temporal appearance variation characteristic function, two spatially varying textures corresponding to the initial and final frames, and two spatially varying textures corresponding to the rates and offsets at each point on the material that determine the evolution of the appearance over time. Our experiments demonstrate that the STAF model can accurately capture appearance variations due to a wide variety of natural processes. Moreover, it enables a number of novel rendering applications, such as the transfer of a time-varying effect to a new material, the control of the evolution of a time-varying process, and the extrapolation of a process beyond what is captured by the acquired data. This project is done in collaboration with Ravi Ramamoorthi and Peter Belhumeur at Columbia University, and Chien-i Tu and Wojciech Matusik at MERL.

Publications

"Time-varying Surface Appearance: Acquisition, Modeling, and Rendering,"
J. Gu, C. Tu, R. Ramamoorthi, P. Belhumeur, W. Matusik and S. K. Nayar,
ACM Trans. on Graphics (also Proc. of ACM SIGGRAPH),
Jul, 2006.
[PDF] [bib] [©]

Images

  Database of Time-Varying Surface Appearance and Factorization:

This image shows the 26 samples acquried in the database and the time-varying process that each sample is subjected to.

     
  Space-Time Appearance and Factorization (STAF):

This image shows the STAF representation of a drying wood sample. The top panel compares the reconstruction of a sample using STAF to the original sample for a chosen lighting and viewing direction. STAF model can also be used for time normalization, wherein the overall appearance changes maintained but the spatial drying patterns are removed. The panel below shows the estimated temporal characterisitic curves for both diffuse and specular parameters. We also show visualizations of the spatial diffuse textures A, D, R and O, along with the normalized initial frame A*f(0)+D and final frame A*f(1)+D. Please see the paper for more details.

     
  STAF Results for More Samples:

This image shows STAF estimation results for 5 samples in the database, each of which has 3 spatial locations marked. The middle row shows the time-varying curves p(x,y,t) (for the red diffuse component) for spatial locations A, B and C. The curves are quite different for the three different points. In the bottom row, we align these time-varying curves using STAF model. The data closely matches the temporal characteristic curve f(t') computed from all the points on the sample.

     
  Examples of Rendering:

This image shows some of our rendering results (please see videos as well). STAF model enables us to manipulate the acquired data. We can create a time-varying texture with larger extent by applying standard 2D texture synthesis to the initial and final frames. Within limits, we have extrapolate the time domain of the original acquired data. We can control the rate and offset terms according to physical laws or user-derfined rules. We also can transfer a time-varying effect to a new material.

     

Videos

If you are having trouble viewing these .avi videos in your browser, please save them to your computer first (by right-clicking and choosing "Save Target As..."), and then open them.

  SIGGRAPH 2006 Video:

This video shows the complete pipeline we have used for our work on time-varying surface appearance, including acqusition, modeling, and rendering. (With narration)

     
  Some Samples in the Database:

This video shows the acquired data for 6 samples in our database for a single lighting and view direction. Our captured data includes a large number of lighting and viewing directions. (With narration)

     
  Rendering:

This video shows 5 rendering examples including texture synthesis, time-extrapolation, control, and transfer. (With narration)

     
  More Synthesis Results:

This video shows texture syntheis examples for the drying rock and the drying wood samples.

     

Slides

SIGGRAPH 2006 presentation     With videos (zip file)

Database

  STAF: Time Varying Appearance Database & Source Code::

The database includes 26 samples, covering a range of phenomena including burning, corrosion, drying on smooth and rough surfaces, and decay. We also include the source code used to process the captured data and estimate the STAF model. For each sample there are three parts of data to download: 6 of the 1280 measurements of the raw data at each time step, the fitted BRDF parameters, and the estimated STAF results. The total size of the database is around 3 GB. Due to its enormous size (about 1,000 Gb), we are not able to release all of the 1280 measurements of the raw data. If you really need it, please send an email to staf@cs.columbia.edu

     

Bidirectional Texture Function

Time-Varying BRDFs

Oren-Nayar Reflectance Model