US 8,026,931 B2Granted

Digital video effects

AssigneeMicrosoft Corporation

InventorsJian Sun, Qiang Wang, Weiwei Zhang, Xiaoou Tang, Heung-Yeung Shum

Life of this patent

Estimating the term…

Loading patent…

Abstract

Digital video effects are described. In one aspect, a foreground object in a video stream is identified. The video stream comprises multiple image frames. The foreground object is modified by rendering a 3-dimensional (3-D) visual feature over the foreground object for presentation to a user in a modified video stream. Pose of the foreground object is tracked in 3-D space across respective ones of the image frames to identify when the foreground object changes position in respective ones of the image frames. Based on this pose tracking, aspect ratio of the 3-D visual feature is adaptively modified and rendered over the foreground object in corresponding image frames for presentation to the user in the modified video stream.

Claims20

Dependent claims

A method at least partially implemented by a computing device, the method comprising:

identifying a foreground object in a video stream comprising multiple image frames;
rendering a three-dimensional (3-D) visual feature over a portion of the foreground object to add a digital video effect to the video stream and generate a modified foreground object;
tracking each pose of the foreground object in 3-D space across the multiple image frames by;
iteratively refining estimations associated with a first and a second conditional distributions in a joint distribution of a Bayesian network that models the foreground object, the first conditional distribution comprises a distribution of a relative pose given correspondences between 3-D model points and two-dimensional (2-D) features of the foreground object, the second conditional distribution comprises a distribution of matching features of the visual object between two frames of the video sequence given the 3-D model points and a relative pose estimation associated with the first conditional distribution;
and
using a Bayesian fusion of the iteratively refined estimations to obtain a current pose of the foreground object, wherein the iteratively refined estimations include an iteratively refined relative pose estimation and an iteratively refined feature matching estimation;
and
maintaining rendered aspect ratios of the 3-D visual feature in correspondence with aspect ratios of a remaining portion of the foreground object as the foreground object changes pose in respective ones of the image frames.

5 more elements

A computing device comprising:

a processor;
and
a memory coupled to the processor, the memory comprising computer-program instructions executable by the processor for:
generating a video stream comprising a 3-D image of a first person involved in a video communication session with a second person using a remote computing device;
separating a foreground object representing the 3-D image from a background of the video stream based on differences between color and contrast attributes of pixels of the foreground object and color and contrast attributes of pixels of the background;
refining boundaries between the foreground object and the background by adaptively attenuating a background contrast while preserving contrasts across the boundaries;
adaptively rendering a 3-D feature over particular ones of multiple video frames that comprise the foreground object in multiple translational and rotational poses to generate a modified video stream, the 3-D feature being rendered over a portion of the 3-D facial features of the first person such that aspect ratios of the 3-D feature are maintained in correspondence with aspect ratios of a remaining portion of 3-D facial features in view of the translational and rotational poses;
and
communicating the modified video stream to the remote computing device for presentation to the second person.

6 more elements

A tangible computer-readable storage medium comprising computer-program instructions executable by a processor, the computer-program instructions, when executed by the processor, for performing operations comprising:

providing a user with one or more video stream background modification options;
presenting the user with one or more costume overlay options;
responsive to selection by the user of a particular background modification option of the video stream background modification options, adaptively modifying background of a video stream using the particular background modification option;
responsive to selection by the user of a particular costume overlay option of the costume overlay options, adaptively rendering a 3-D visual feature associated with the costume overlay option over a portion of a foreground object in frames that comprise the video stream, the portions being parts of facial features;
maintaining aspect ratios of the 3-D visual features in correspondence with aspect ratios of a remaining portion of the foreground object as the foreground object changes translational or rotational position in the frames;
and
communicating the video stream to a remote computing device for presentation to a different user.

4 more elements

Dependencies are inferred from the claim text.