# US 2023/0410397 A1 · Physics-based image generation using one or more neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410397 A1
- Document ID: US20230410397A1
- Status: Published application
- Kind: A1
- Application number: 17/842,481
- Filed: 2022-06-16
- Published: 2023-12-21
- Earliest claimed: 2022-06-16
- Assignees: NVIDIA Corporation
- Applicants: Nvidia Corporation
- Inventors: Tingwu Wang; Yunrong Guo; Cheng Xie; Xue Bin Peng; Sanja Fidler
- Web page: https://patentgrep.com/patent/US20230410397A1

## CPC

- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/042: Knowledge-based neural networks; Logical representations of neural networks
- G06N 3/094: Adversarial learning
- G06T 17/10: Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
- G06N 3/063: using electronic means
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Apparatuses, systems, and techniques are presented to generate images representing realistic motion or activity. In at least one embodiment, one or more neural networks are used to generate one or more images of one or more characters performing one or more actions based, at least in part, upon one or more physical capabilities of the one or more characters.

## Claims

### Claim 1

A processor, comprising:
one or more circuits to use one or more neural networks to generate one or more images of one or more characters performing one or more actions based, at least in part, upon one or more physical capabilities of the one or more characters.

### Claim 2 (depends on 1)

The processor of claim 1, wherein the one or more circuits are further to represent the one or more characters using one or more character graphs, each character graph including a connected set of nodes representing body components of a respective character having corresponding parameter values for the one or more physical capabilities.

### Claim 3 (depends on 2)

The processor of claim 2, wherein the one or more circuits are further to determine motions for the connected nodes of the one or more character graphs based, at least in part, upon current state and target state data for the one or more actions.

### Claim 4 (depends on 3)

The processor of claim 3, wherein the one or more circuits are further to utilize one or more graph neural networks to update state information for the connected nodes by exchanging state information between the connected nodes for each of a set of forward passes through the one or more graph neural networks.

### Claim 5 (depends on 2)

The processor of claim 2, wherein the one or more circuits are further to generate different character graphs for different characters having different morphologies or bodily structures.

### Claim 6 (depends on 1)

The processor of claim 1, wherein the one or more circuits are further to generate the one or more images in a sequence to produce physics-based animation for the one or more characters.

### Claim 7

A system comprising:
one or more processors to generate one or more images of one or more characters performing one or more actions based, at least in part, upon one or more physical capabilities of the one or more characters.

### Claim 8 (depends on 7)

The system of claim 7, wherein the one or more processors are further to represent the one or more characters using one or more character graphs, each character graph including a connected set of nodes representing body components of a respective character having corresponding parameter values for the one or more physical capabilities.

### Claim 9 (depends on 8)

The system of claim 8, wherein the one or more processors are further to determine motions for the connected nodes of the one or more character graphs based, at least in part, upon current state and target state data for the one or more actions.

### Claim 10 (depends on 9)

The system of claim 9, wherein the one or more processors are further to utilize one or more graph neural networks to update state information for the connected nodes by exchanging state information between the connected nodes for each of a set of forward passes through the one or more graph neural networks.

### Claim 11 (depends on 8)

The system of claim 8, wherein the one or more processors are further to generate different character graphs for different characters having different morphologies or bodily structures.

### Claim 12 (depends on 7)

The system of claim 7, generate the one or more images in a sequence to produce physics-based animation for the one or more characters.

### Claim 13

A method comprising:
generating one or more images of one or more characters performing one or more actions based, at least in part, upon one or more physical capabilities of the one or more characters.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
representing the one or more characters using one or more character graphs, each character graph including a connected set of nodes representing body components of a respective character having corresponding parameter values for the one or more physical capabilities.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
determining motions for the connected nodes of the one or more character graphs based, at least in part, upon current state and target state data for the one or more actions.

### Claim 16 (depends on 15)

The method of claim 15, further comprising:
utilizing one or more graph neural networks to update state information for the connected nodes by exchanging state information between the connected nodes for each of a set of forward passes through the one or more graph neural networks.

### Claim 17 (depends on 14)

The method of claim 14, further comprising:
generating different character graphs for different characters having different morphologies or bodily structures.

### Claim 18 (depends on 13)

The method of claim 13, further comprising:
generating the one or more images in a sequence to produce physics-based animation for the one or more characters.

### Claim 19

A machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least:
generate one or more images of one or more characters performing one or more actions based, at least in part, upon one or more physical capabilities of the one or more characters.

### Claim 20 (depends on 19)

The machine-readable medium of claim 19, wherein the instructions if performed further cause the one or more processors to represent the one or more characters using one or more character graphs, each character graph including a connected set of nodes representing body components of a respective character having corresponding parameter values for the one or more physical capabilities.

### Claim 21 (depends on 20)

The machine-readable medium of claim 20, wherein the instructions if performed further cause the one or more processors to:
determine motions for the connected nodes of the one or more character graphs based, at least in part, upon current state and target state data for the one or more actions.

### Claim 22 (depends on 21)

The machine-readable medium of claim 21, wherein the instructions if performed further cause the one or more processors to:
utilize one or more graph neural networks to update state information for the connected nodes by exchanging state information between the connected nodes for each of a set of forward passes through the one or more graph neural networks.

### Claim 23 (depends on 20)

The machine-readable medium of claim 20, wherein the instructions if performed further cause the one or more processors to:
generate different character graphs for different characters having different morphologies or bodily structures.

### Claim 24 (depends on 19)

The machine-readable medium of claim 19, wherein the instructions if performed further cause the one or more processors to:
generate the one or more images in a sequence to produce physics-based animation for the one or more characters.

### Claim 25

An image generation system, comprising:
one or more processors to generate one or more images of one or more characters performing one or more actions based, at least in part, upon one or more physical capabilities of the one or more characters; and
memory for storing network parameters for the one or more neural networks.

### Claim 26 (depends on 25)

The image generation system of claim 25, wherein the one or more processors are further to represent the one or more characters using one or more character graphs, each character graph including a connected set of nodes representing body components of a respective character having corresponding parameter values for the one or more physical capabilities.

### Claim 27 (depends on 26)

The image generation system of claim 26, wherein the one or more processors are further to determine motions for the connected nodes of the one or more character graphs based, at least in part, upon current state and target state data for the one or more actions.

### Claim 28 (depends on 27)

The image generation system of claim 27, wherein the one or more processors are further to utilize one or more graph neural networks to update state information for the connected nodes by exchanging state information between the connected nodes for each of a set of forward passes through the one or more graph neural networks.

### Claim 29 (depends on 28)

The image generation system of claim 28, wherein the one or more processors are further to generate different character graphs for different characters having different morphologies or bodily structures.

### Claim 30 (depends on 25)

The image generation system of claim 25, wherein the one or more processors are further to generate the one or more images in a sequence to produce physics-based animation for the one or more characters.
