US 6,611,268 B1Granted

System and process for generating 3D video textures using video-based rendering techniques

AssigneeMicrosoft Corporation

InventorsRichard S. Szeliski, David Salesin, Arno Schödl

Life of this patent

Estimating the term…

Loading patent…

Abstract

A system and process for generating a 3D video animation of an object referred to as a 3D Video Texture is presented. The 3D Video Texture is constructed by first simultaneously videotaping an object from two or more different cameras positioned at different locations. Video from, one of the cameras is used to extract, analyze and synthesize a video sprite of the object of interest. In addition, the first, contemporaneous, frames captured by at least two of the cameras are used to estimate a 3D depth map of the scene. The background of the scene contained within the depth map is then masked out, and a clear shot of the scene background taken before filming of the object began, leaving just the object. To generate each new frame in the 3D video animation, the extracted region making up a “frame” of the video sprite is mapped onto the previously generated 3D surface. The-resulting image is rendered from a novel viewpoint, and then combined with a flat image of the background which has been warped to the correct location. In cases where it is anticipated that the subject could move frequently, the foregoing part of the procedure associated with estimating a 3D depth map of the scene and extracting the 3D surface representation of the object is performed for each subsequent set of contemporaneous frames captured by at least two of the cameras.

Claims4

Dependent claims

A computer-implemented process for generating a 3D video animation of an object, comprising using a computer to perform the following process actions.

(a) simultaneously capturing images of a scene containing the object from two or more different cameras positioned at different locations, wherein at least one of the cameras is a video camera which captures sequential images of the object to generate an input video clip;
(b) estimating a 3D depth map of the scene containing the object using a first, contemporaneous, image captured by at least two of the cameras;
(c) extracting the region of the 3D depth map representing the object;
(d) extracting a corresponding region of each of a selected number of frames of the input video clip which depicts the object;
(e) computing a value indicative of the similarity between the extracted region of each of the selected frames and every corresponding extracted region of the other selected frames;
(f) for each extracted region, analyzing the similarity values associated therewith to identify potentially acceptable transitions between it and corresponding extracted regions of the other selected frames;
(g) synthesizing a new video sequence from the group of corresponding extracted regions by using the identified transitions to generate an order in which the extracted regions associated with these transitions are to be played;
(h) mapping onto the extracted region of the 3D depth map, the extracted region of the new video sequence that was specified as first to be played to produce a 3D image of the object;
(i) rendering the 3D image of the object from a prescribed novel viewpoint;
(j) combining the 3D image of the object with a background image to create a frame of the 3D video animation;
and
(k) repeating process actions (h) through (j) for each subsequent extracted region of the new video sequence in the order specified in the synthesizing process action, to create each subsequent frame of the 3D video animation.

9 more elements

A computer-implemented process for generating a 3D video animation of an object, comprising using a computer to perform the following process actions:

(a) simultaneously capturing images of a scene containing the object from two or more different cameras positioned at different locations, wherein at least one of the cameras is a video camera which captures sequential images of the object to generate an input video clip;
(b) estimating a 3D depth map of the scene containing the object using contemporaneous images captured by at least two of the cameras;
(c) extracting the region of the 3D depth map representing the object;
(d) extracting a corresponding region of each of a selected number of frames of the input video clip which depicts the object;
(e) computing a value indicative of the similarity between the extracted region of each of the selected frames and every corresponding extracted region of the other selected frames;
(f) for each extracted region, analyzing the similarity values associated therewith to identify potentially acceptable transitions between, it and corresponding extracted regions of the other selected frames;
(g) synthesizing a new video sequence from the group of corresponding extracted regions by using the identified transitions to generate an order in which the extracted regions associated with these transitions are to be played;
(h) mapping onto the extracted region of the 3D depth map, the extracted region of the new video sequence that was specified as first to be played to produce a 3D image of the object;
(i) rendering the 3D image of the object from a prescribed novel viewpoint;
(j) combining the 3D image of the object with a background image to create a frame of the 3D video animation;
and
(k) repeating process actions (b) and (c) for each subsequent, temporally-corresponding, frames of said at least two of the video cameras, and repeating process actions (h) through (j) for each subsequent extracted region of the new video sequence in the order specified in the synthesizing process action, to create each subsequent frame of the 3D video animation.

9 more elements

Dependencies are inferred from the claim text.