# US 2024/0257470 A1 · Avatar personalization using image generation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0257470 A1
- Document ID: US20240257470A1
- Status: Published application
- Kind: A1
- Application number: 18/458,731
- Filed: 2023-08-30
- Published: 2024-08-01
- Earliest claimed: 2023-01-30
- Assignees: Meta Platforms Technologies, LLC
- Applicants: Meta Platforms Technologies, LLC
- Inventors: Sonal Gupta; Samaneh Azadi; Mian Akbar Shah; Thomas Falstad Hayes; Devi Niru Parikh
- Web page: https://patentgrep.com/patent/US20240257470A1

## CPC

- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- A63F 13/52: involving aspects of the displayed game scene
- A63F 13/55: Controlling game characters or game objects based on the game progress
- A63F 13/63: by the player, e.g. authoring using a level editor
- A63F 13/67: adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method and system for personalized avatar generation. The method includes receiving a text prompt and generating a first pose based on the text prompt using a first model. The method also includes re-targeting the first pose to a target avatar body. The method also includes identifying a predefined avatar configuration corresponding to a user based on a profile of the user. The method also includes converting the target avatar body by applying the predefined avatar configuration to the target avatar body. The method also includes rendering an avatar, using a second model, based on the target avatar body with the predefined avatar configuration, wherein the avatar is in the first pose.

## Claims

### Claim 1

A computer-implemented method, performed by at least one processor, for personalized avatar generation, the method comprising:
receiving a text prompt;
generating a first pose based on the text prompt using a first model;
re-targeting the first pose to a target avatar body;
identifying a predefined avatar configuration corresponding to a user based on a profile of the user;
converting the target avatar body by applying the predefined avatar configuration to the target avatar body; and
rendering an avatar, using a second model, based on the target avatar body with the predefined avatar configuration, wherein the avatar is in the first pose.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the text prompt describes a scene and interactions of the avatar within the scene.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the first pose is a 3D pose generated based on human joint and limb orientation and positioning parameters.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the first pose is a human body pose represented by SMPL parameters.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, the generating the first pose further comprising:
extracting first text embeddings from the text prompt;
mapping the first text embeddings to second text embeddings retrieved from a dataset;
selecting one or more second poses corresponding to the second text embeddings; and
determining the first pose based on the one or more second poses.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, further comprising:
training the first model on a dataset including body poses and corresponding text descriptions, the training comprising:
identifying humans in images of the dataset,
segmenting the humans from the images, and
extracting 3D Skinned Multi-Person Linear Model (SMPL) annotations from segmented humans from the images.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the target avatar body is a gray avatar-human body representation.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the re-targeting comprises matching corresponding joints, in position and orientation, from the first pose and the target avatar body.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, further comprising generating an image, using the second model, of a scene in a virtual environment including the avatar interacting with objects in the scene.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein the second model performs conditional stable diffusion inpainting to generate the image by outpainting from the avatar to fill in the scene and objects in the scene, and the second model is conditioned on at least the avatar and the text prompt.

### Claim 11

A system for personalized avatar generation, the system comprising:
one or more processors; and
a memory storing instructions which, when executed by the one or more processors, cause the system to:
receive a text prompt describing a scene and interactions of the avatar within the scene;
generate a first pose based on the text prompt using a first model;
re-target the first pose to a target avatar body;
identify a predefined avatar configuration corresponding to a user based on a profile of the user;
convert the target avatar body by applying the predefined avatar configuration to the target avatar body; and
render an avatar, using a second model, based on the target avatar body with the predefined avatar configuration, wherein the avatar is in the first pose.

### Claim 12 (depends on 11)

The system of claim 11, wherein the first pose is a 3D pose generated based on human joint and limb orientation and positioning parameters.

### Claim 13 (depends on 11)

The system of claim 11, wherein the first pose is a human body pose represented by SMPL parameters.

### Claim 14 (depends on 11)

The system of claim 11, wherein the one or more processors further execute instructions to:
extract first text embeddings from the text prompt;
map the first text embeddings to second text embeddings retrieved from a dataset;
select one or more second poses corresponding to the second text embeddings; and
determine the first pose based on the one or more second poses.

### Claim 15 (depends on 11)

The system of claim 11, wherein the one or more processors further execute instructions to train the first model on a dataset including body poses and corresponding text descriptions, the instructions causing the system to:
identify humans in images of the dataset;
segment the humans from the images; and
extract 3D Skinned Multi-Person Linear Model (SMPL) annotations from segmented humans from the images.

### Claim 16 (depends on 11)

The system of claim 11, wherein the target avatar body is a gray avatar-human body representation.

### Claim 17 (depends on 11)

The system of claim 11, wherein the one or more processors further execute instructions to match corresponding joints, in position and orientation, from the first pose and the target avatar body.

### Claim 18 (depends on 11)

The system of claim 11, wherein the one or more processors further execute instructions to:
generate an image, using the second model, of a scene in a virtual environment including the avatar interacting with objects in the scene.

### Claim 19 (depends on 11)

The system of claim 11, wherein the second model performs conditional stable diffusion inpainting to generate an image by outpainting from the avatar to fill in the scene and objects in the scene, and the second model is conditioned on at least the avatar and the text prompt.

### Claim 20

A non-transient computer-readable storage medium having instructions embodied thereon, the instructions being executable by one or more processors to perform a method for personalized avatar generation and cause the one or more processors to:
receive a text input describing an avatar in a scene;
generating a body pose based on the text prompt;
re-targeting the body pose to a target avatar body;
generating a personalized avatar based on a predefined avatar configuration being applied to the target avatar body; and
generating an image of the avatar in the scene based on the personalized avatar, the image including the avatar being in the body pose with the predefined avatar configuration.
