US 11,238,634 B2 · Motion model refinement based on contact analysis and optimization

Abstract

In some embodiments, a motion model refinement system receives an input video depicting a human character and an initial motion model describing motions of individual joint points of the human character in a three-dimensional space. The motion model refinement system identifies foot joint points of the human character that are in contact with a ground plane using a trained contact estimation model. The motion model refinement system determines the ground plane based on the foot joint points and the initial motion model and constructs an optimization problem for refining the initial motion model. The optimization problem minimizes the difference between the refined motion model and the initial motion model under a set of plausibility constraints including constraints on the contact foot joint points and a time-dependent inertia tensor-based constraint. The motion model refinement system obtains the refined motion model by solving the optimization problem.

Claims20

Dependent claims

A computer-implemented method in which one or more processing devices perform operations comprising:

determining contact joint points for a human character contained in a sequence of video frames using a trained contact estimation model, wherein the contact joint points are determined by the trained contact estimation model as being in contact with a ground plane;
determining the ground plane, in a three-dimensional space defined by an initial motion model of the human character, based on the contact joint points and the initial motion model, wherein the initial motion model of the human character is estimated from the sequence of video frames and describes motions of joint points of the human character in the three-dimensional space;
determining a refined motion model over the initial motion model by performing an optimization under a set of constraints defined based on the initial motion model, the determined ground plane, and the contact joint points;
and
applying the refined motion model on a target computer-generated character.

2 more elements

A system comprising:

a processing device;
and
a non-transitory computer-readable medium having program code that is stored thereon, the program code executable by the processing device for performing operations, comprising:
generating an input vector for a sequence of video frames by determining positions of joint points of a character contained in the sequence of video frames;
executing a trained neural network based on the input vector to generate indications of contact statuses of a subset of the joint points with a ground plane;
estimating the ground plane in a three-dimensional space based on joint points that are in contact with the ground plane according to the indications of the contact statuses and an initial motion model of the character in the three-dimensional space extracted from the sequence of video frames;
determining a refined motion model based on the initial motion model by performing an optimization under a set of constraints;
and
applying the refined motion model on a target computer-generated character to generate a computer animated character.

6 more elements

A non-transitory computer-readable medium having program code that is stored thereon, the program code executable by one or more processing devices for performing operations comprising:

determining contact joint points for a human character contained in a sequence of video frames using a trained contact estimation model, wherein the contact joint points are determined by the trained contact estimation model as being in contact with a ground plane;
determining the ground plane, in a three-dimensional space defined by an initial motion model of the human character, based on the contact joint points and the initial motion model, wherein the initial motion model of the human character is estimated from the sequence of video frames and describes motions of joint points of the human character in the three-dimensional space;
determining a refined motion model over the initial motion model by performing an optimization under a set of constraints defined based on the initial motion model, the determined ground plane, and the contact joint points;
and
applying the refined motion model on a target computer-generated character.

2 more elements

Dependencies are inferred from the claim text.

Motion model refinement based on contact analysis and optimization

Life of this patent

Motion model refinement based on contact analysis and optimization

Life of this patent

Abstract

Claims20