US 11,887,231 B2Granted

Avatar animation system

AssigneeTahoe Research, Ltd.

InventorsMinje Park, Tae-Hoon Kim, Myung-Ho Ju, Jihyeon Yi, Xiaolu Shen, Lidan Zhang, Qiang Li

Life of this patent

Estimating the term…

Loading patent…

Abstract

Avatar animation systems disclosed herein provide high quality, real-time avatar animation that is based on the varying countenance of a human face. In some example embodiments, the real-time provision of high quality avatar animation is enabled, at least in part, by a multi-frame regressor that is configured to map information descriptive of facial expressions depicted in two or more images to information descriptive of a single avatar blend shape. The two or more images may be temporally sequential images. This multi-frame regressor implements a machine learning component that generates the high quality avatar animation from information descriptive of a subject's face and/or information descriptive of avatar animation frames previously generated by the multi-frame regressor. The machine learning component may be trained using a set of training images that depict human facial expressions and avatar animation authored by professional animators to reflect facial expressions depicted in the set of training images.

Claims19

Dependent claims

An avatar animation system comprising:

a display;
a camera;
and
at least one processor communicatively coupled to the display and the camera, the at least one processor being configured to:
generate a plurality of meshes based on a plurality of images of a face of a subject generated by the camera, respective ones of the plurality of meshes corresponding to respective ones of the plurality of images of the face of the subject, the plurality of meshes including:
a first set of meshes, respective ones of the first set of meshes corresponding to respective ones of a first set of the images of the face of the subject, and
a second set of meshes, respective ones of the second set of meshes corresponding to respective ones of a second set of the images of the face of the subject,
wherein the first set of meshes is not identical to the second set of meshes, and an entirety of at least one mesh from the plurality of meshes is included in both the first set of meshes and the second set of meshes;
generate a first set of blend shape weights for a first frame of avatar animation, the first set of blend shape weights based on the first set of meshes;
generate a second set of blend shape weights for a second frame of avatar animation, the second frame of avatar animation being different from the first frame of avatar animation, and the second set of blend shape weights being based on at least:
the second set of meshes and the first set of blend shape weights;
generate the second frame of avatar animation based on the second set of blend shape weights;
and
cause the display to present the second frame of avatar animation.

11 more elements

A method comprising:

generating a plurality of meshes based on a plurality of images of a face of a subject, respective ones of the plurality of meshes corresponding to respective ones of the plurality of images of the face of the subject, the plurality of meshes including:
a first set of meshes, respective ones of the first set of meshes corresponding to respective ones of a first set of the images of the face of the subject, and
a second set of meshes, respective ones of the second set of meshes corresponding to respective ones of a second set of the images of the face of the subject,
wherein the first set of meshes is not identical to the second set of meshes, and an entirety of at least one mesh from the plurality of meshes is included in both the first set of meshes and the second set of meshes;
generating a first set of blend shape weights for a first frame of avatar animation using feedback, the first set of blend shape weights based on the first set of meshes;
generating a second set of blend shape weights for a second frame of avatar animation, the second frame of avatar animation being different from the first frame of avatar animation, and the second set of blend shape weights being based on at least:
the second set of meshes and the first set of blend shape weights;
and
generating the second frame of avatar animation based on the second set of blend shape weights.

6 more elements

At least one storage device comprising instructions that, when executed, cause one or more processors to:

generate a plurality of meshes based on a plurality of images of a face of a subject, respective ones of the plurality of meshes corresponding to respective ones of the plurality of images of the face of the subject, the plurality of meshes including:
a first set of meshes, respective ones of the first set of meshes corresponding to respective ones of a first set of the images of the face of the subject, and
a second set of meshes, respective ones of the second set of meshes corresponding to respective ones of a second set of the images of the face of the subject,
wherein the first set of meshes is not identical to the second set of meshes, and an entirety of at least one mesh from the plurality of meshes is included in both the first set of meshes and the second set of meshes;
generate a first set of blend shape weights for a first frame of avatar animation, the first set of blend shape weights based on the first set of meshes;
generate a second set of blend shape weights for a second frame of avatar animation, the second frame of avatar animation being different from the first frame of avatar animation, and the second set of blend shape weights being based on at least:
the second set of meshes and the first set of blend shape weights;
and
generate the second frame of avatar animation based on the second set of blend shape weights.

6 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/655,686Filed 17 Oct 2019 · continuation
Published: US 2020/0051306 A113 Feb 2020
Granted: 30 Jan 2024
Term ends: Not estimatedThe source records don’t support an estimate.

Classification7 codes

G06T 13/40
Three-dimensional [3D] animation: of characters, e.g. humans, animals or virtual beings
G06T 7/73
Determining position or orientation of objects or cameras: using feature-based methods
G06T 17/20
Three-dimensional [3D] modelling for computer graphics: finite element generation, e.g. wire-frame surface description, tesselation
G06T 2207/20084
Special algorithmic details: artificial neural networks [ANN]

Family

PCT/CN2015/097826Parent of this continuation · filed 18 Dec 2015
US 2020/0051306 A1Pre-grant publication · published 13 Feb 2020
Application15/124,811Parent of this continuation
US 10,475,225 B2Parent of this continuation
US 2017/0256086 A1Published application
US 11,887,231 B2This patent

From the US filing history. Foreign family members aren’t listed.