In this example, we demonstrate the importance of our color and lighting adjustment algorithm. We replace (a) the face in the input photograph with (b) the face selected from the library. Replacement results (c) without and (d) with recoloring and relighting. Notice the significantly improved realism in the final result.

Here, we show several examples of the results obtained using our system. Each example shows, in order from left to right, the input face image, a candidate face, and the replacement result. Note the realism of the results, despite differences in pose, lighting and facial appearance. Each row contains (from left to right) the original photograph, a candidate face selected from the library, and the replacement result produced automatically using our algorithm. The age and gender mismatches in (c) and (d) could be avoided by enforcing consistency across those attributes (which our system does not currently do).

To preserve privacy in online collections of photos, one can use our system to automatically replace each face in an input image with the top-ranked candidate taken from a collection of stock photographs. We show the result of automatically replacing the input faces (top) with the top-ranked candidate from the face library to obtain the de-identified results (bottom). No user intervention was used to produce this result.

As a special case of face de-identification (or for use as a special effect), we can limit the system to use candidates only within the same image, resulting in the switching of faces. Here, we show the result of switching Elvis Presley and Richard Nixon’s faces (left) with each other to obtain the de-identified output (right).

When taking group photographs, it is often difficult to get a “perfect” picture – where, for example, everyone is smiling, with eyes open, and looking at the camera. From a set of images taken using the “burst” mode of a camera (left panel), a composite image is created in which everyone is smiling and has their eyes open (right panel). The candidate faces for each child are constrained by the relative positions of the faces in all images, and thus no face recognition is required. While in this case the best replacement face for each child was selected manually (outlined in blue), blink and smile detection could be applied to select them automatically.

Here, we show several examples of limitations of our algorithm itself.Input and replacement candidate faces are shown on the top, replacement results in the middle, and a detailed inset of the problem area on the bottom. The lack of eyeglasses in (a) and the occluding finger in (b) cause visual artifacts in the results. In (c), the extreme pose of the face results in it being blended into the background. These problems could be solved by dynamically selecting optimal replacement regions. (d) shows a relighting failure case, caused by forcing a replacement between images with very different lighting (skipping our lighting selection step).

This video introduces the complete system for automatic face replacement in images, summarizes the face replacement algorithm, and demonstrates several applications of our method, including face de-identification, personalized face replacement, and the creation of appealing group photographs from a set of images. (With narration)