# US 2021/0343059 A1 · Motion Retargeting with Kinematic Constraints

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0343059 A1
- Document ID: US20210343059A1
- Status: Published application
- Kind: A1
- Application number: 16/864,724
- Filed: 2020-05-01
- Published: 2021-11-04
- Earliest claimed: 2020-05-01
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Ruben Eduardo Villegas; Jun Saito; Jimei Yang; Duygu Ceylan Aksit
- Web page: https://patentgrep.com/patent/US20210343059A1

## CPC

- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06T 15/20: Perspective computation
- G06T 15/40: Hidden part removal
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Motion retargeting with kinematic constraints is implemented in a digital medium environment. Generally, the described techniques provide for retargeting motion data from a source motion sequence to a target visual object. Accordingly, the described techniques position a target visual object in a defined visual environment to identify kinematic constraints of the target object relative to the visual environment. Further, the described techniques utilize an iterative optimization process that fine tunes the conformance of retargeted motion of a target object to the identified kinematic constraints.

## Claims

### Claim 1

A method for retargeting an animation implemented by at least one computing device, the method comprising:
processing, by the at least one computing device, a source motion object by inputting the source motion object to a neural network to generate target joint coordinates for joints of a target visual object, and a root velocity for a root joint of the target visual object;
positioning an animation of the target visual object, by the at least one computing device, by:
positioning the target visual object in a visual space by positioning, based on the root velocity, the joint coordinates within the visual space to identify one or more kinematic constraints of the target visual object relative to the visual space; and
optimizing animation of the target visual object by inputting the root velocity into the neural network to conform the animation of the target visual object to the one or more kinematic constraints; and
implementing, by the at least one computing device, a supervision operation for tuning animation of the target visual object, the supervision operation utilizing root motion data from the source motion object as scaled based on a ratio of a leg height of the source visual object, and a leg height of the target visual object.

### Claim 2 (depends on 1)

The method as described in claim 1, wherein the target visual object includes multiple structural members, the joints represents connection points for the structural members, and the root joint comprises a reference point for one or more other joints of the target visual object.

### Claim 3 (depends on 1)

The method as described in claim 1, wherein said processing comprises:
inputting the source motion data for the source motion object to an encoder recurrent neural network to generate encoded motion data;
inputting the encoded motion data to a decoder recurrent neural network to generate decoded motion data including joint rotations for the joints of a target visual object, and root motion for a root position of the target visual object; and
inputting the joint rotations for the joints of a target visual object to a forward kinematics module to generate the joint coordinates for the joints of the target visual object.

### Claim 4 (depends on 1)

The method as described in claim 1, wherein the visual space comprises a coordinate space, and said positioning comprises positioning the joint coordinates in the visual space based on corresponding coordinates of the visual space.

### Claim 5 (depends on 1)

The method as described in claim 1, wherein the kinematic constraints are based on contact of one or more structural members of the target visual object with one or more surfaces of the visual space.

### Claim 6 (depends on 5)

The method as described in claim 5, wherein the one or more surfaces of the visual space comprise a ground surface.

### Claim 7

(canceled)

### Claim 8

(canceled)

### Claim 9 (depends on 1)

The method as described in claim 1, wherein said optimizing animation of the target visual object comprising backpropagating the root velocity through the neural network.

### Claim 10

A system for retargeting animation, the system comprising:
a retargeting model implemented at least partially in hardware of at least one computing device to take a source motion data from a source motion object as input and output target animation data to be applied to animate a target object, the target animation data including joint coordinates for joints of a target visual object, and a root velocity for a root joint of the target visual object;
a positioning module implemented at least partially in the hardware of the at least one computing device to position the target visual object in a visual space by positioning, based on the root velocity, the joint coordinates within the visual space to identify one or more kinematic constraints of the target visual object relative to the visual space;
an optimization module implemented at least partially in the hardware of the at least one computing device to optimize animation of the target visual object by inputting the root velocity into the retargeting module to conform the animation of the target visual object to the one or more kinematic constraints; and
a training module implemented to use joint rotation data and root motion data from the source motion data to perform supervision to fine tune the animation of the target visual object, wherein the training module is implemented to use the root motion data as scaled based on a ratio of a leg height of the source visual object animated in the source motion data, and a leg height of the target visual object.

### Claim 11 (depends on 10)

The system as described in claim 10, wherein the retargeting model comprises:
an encoder recurrent neural network to take the source motion data as input to generate encoded motion data;
a decoder recurrent neural network to take the encoded motion data and joint coordinates for the target visual object as input, and generate decoded motion data including the root velocity for the root joint of the target visual object and joint rotations for the joints of the target visual object; and
a forward kinematics module to take the joint rotations as input and generate the joint coordinates for the joints of a target visual object.

### Claim 12 (depends on 10)

The system as described in claim 10, wherein the visual space comprises a coordinate space, and wherein said positioning module is implemented to position the joint coordinates in the visual space based on corresponding coordinates of the visual space.

### Claim 13 (depends on 10)

The system as described in claim 10, wherein the kinematic constraints are based on contact of the target visual object with one or more surfaces of the visual space.

### Claim 14 (depends on 10)

The system as described in claim 10, wherein the optimization module is implemented to optimize animation of the target visual object animation of the target by backpropagating the root velocity through the neural network.

### Claim 15

(canceled)

### Claim 16

(canceled)

### Claim 17

A system for retargeting animation, the system comprising:
one or more processors; and
one or more computer-readable storage media storing instructions that are executable by the one or more processors to:
invoke a neural network to take a source motion sequence as input and output target animation data to be applied to animate a target object, the target animation data including joint coordinates for joints of a target visual object, and a root velocity for a root joint of the target visual object;
position the target visual object in a visual space by positioning, based on the root velocity, the joint coordinates within the visual space to identify one or more kinematic constraints of the target visual object relative to the visual space;
optimize animation of the target visual object by inputting the root velocity into the neural network to conform the animation of the target visual object to the one or more kinematic constraints; and
implement a supervision operation for tuning animation of the target visual object including to utilize as part of the supervision operation root motion data from a source motion object represented in the source motion sequence as scaled based on a ratio of a leg height of the source visual object, and a leg height of the target visual object.

### Claim 18 (depends on 17)

The system of claim 17, wherein the one or more kinematic constraints comprise contact of the target visual object with a boundary of the visual space.

### Claim 19 (depends on 17)

The system of claim 17, wherein the visual space comprises a coordinate space, and wherein said positioning module is implemented to position the joint coordinates in the visual space based on corresponding coordinates of the visual space.

### Claim 20 (depends on 17)

The system of claim 17, wherein the instructions are further executable by the one or more processors to train the neural network using joint rotation data and the root motion data from the source motion object.

### Claim 21 (depends on 1)

The method as described in claim 1, wherein said optimizing comprises backpropagating the root velocity through a decoder of the neural network to cause the root joint to move globally within the visual space toward one or more constraint points defined by the one or more kinematic constraints.

### Claim 22 (depends on 21)

The method as described in claim 21, wherein said optimizing further comprises applying the one or more kinematic constraints to the backpropagated root velocity.

### Claim 23 (depends on 10)

The system as described in claim 10, wherein the optimization module is implemented to optimize animation of the target visual object by backpropagating the root velocity through a decoder of the retargeting model to cause the root joint to move globally within the visual space toward one or more constraint points defined by the one or more kinematic constraints.

### Claim 24 (depends on 17)

The system of claim 17, wherein to optimize animation of the target visual object includes to backpropagate the root velocity through a decoder of the neural network to cause the root joint to move globally within the visual space toward one or more constraint points defined by the one or more kinematic constraints.
