# US 2021/0275925 A1 · Virtual character generation from image or video data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0275925 A1
- Document ID: US20210275925A1
- Status: Published application
- Kind: A1
- Application number: 17/207,408
- Filed: 2021-03-19
- Published: 2021-09-09
- Earliest claimed: 2019-03-27
- Assignees: Electronic Arts Inc.
- Applicants: Electronic Arts Inc.
- Inventors: John Kolen; Harold Henry Chaput; Navid Aghdaie; Kazi Atif-Uz Zaman; Kenneth Alan Moss
- Web page: https://patentgrep.com/patent/US20210275925A1

## CPC

- A63F 13/655: by importing photos, e.g. of the player
- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/40: Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
- A63F 13/428: involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
- A63F 13/63: by the player, e.g. authoring using a level editor
- A63F 13/67: adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/20: Ensemble learning
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G06V 20/20: in augmented reality scenes
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- A63F 2300/5553: user representation in the game field, e.g. avatar
- G06N 3/045: Combinations of networks
- G06N 20/00: Machine learning
- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06T 2210/16: Cloth
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods for generating a customized virtual character are disclosed. A system may obtain video data or other media depicting a real person, and then may provide the obtained media to one or more machine learning models configured to learn visual appearance and behavior information regarding the particular person depicted in the video or other media. The system may then generate a custom visual appearance model and a custom behavior model corresponding to the real person, which may subsequently be used to render, within a virtual environment of a video game, a virtual character that resembles the real person in appearance and in-game behavior.

## Claims

### Claim 1

A system comprising:
a data store that stores three-dimensional (3D) model data for virtual characters of a video game, wherein the 3D model data includes a human base model, wherein the human base model comprises 3D mesh data defining body shape of a generic human figure; and
a computing system in electronic communication with the data store and configured to execute computer-readable instructions that configure the computing system to:
obtain input media depicting a real person, wherein the input media comprises at least a video recording of the real person as captured by a camera;
provide at least a portion of the input media to a machine learning model configured to extract visual information regarding one or more humans depicted in image or video data;
alter the 3D mesh data of the human base model based on visual information extracted by the machine learning model to generate custom 3D model data corresponding to the real person;
extract, from the input media, visual information regarding a first item worn by the real person as depicted in the input media, wherein the first item comprises a clothing item or an accessory;
generate a virtual item corresponding to the first item worn by the real person, wherein the virtual item includes a texture generated based on the visual information; and
render, within a 3D virtual environment of the video game, a series of frames for display that depict a virtual character that resembles the real person performing one or more actions, wherein a visual appearance of the virtual character as rendered is based at least in part on the custom 3D model data and includes depiction of the virtual character wearing the virtual item.

### Claim 2

(canceled)

### Claim 3 (depends on 1)

The system of claim 1, wherein the computer-readable instructions further configure the computing system to generate a virtual clothing selection interface for display to a user, wherein the virtual clothing selection interface identifies a subset of virtual clothing items from a catalog that are available for application to the virtual character, wherein the subset of virtual clothing items are selected from the catalog by the computing system based at least in part on one or more clothing styles of the real person as determined from an automated analysis of the input media.

### Claim 4 (depends on 1)

The system of claim 1, wherein the computer-readable instructions further configure the computing system to:
store, in the data store, custom virtual character data corresponding to the real person, wherein the custom virtual character data includes at least the custom 3D model.

### Claim 5 (depends on 4)

The system of claim 4, wherein the computer-readable instructions further configure the computing system to:
provide the custom virtual character data to two or more video game applications for incorporation of the virtual character into virtual worlds associated with each of the two or more video game applications.

### Claim 6 (depends on 1)

The system of claim 1, wherein the custom 3D model data corresponding to the real person comprises a custom texture extracted from the input media.

### Claim 7 (depends on 1)

The system of claim 1, wherein the machine learning model comprises a generative adversarial network that includes a generator and a discriminator.

### Claim 8 (depends on 1)

The system of claim 1, wherein the machine learning model comprises a convolutional neural network.

### Claim 9

(canceled)

### Claim 10

A computer-implemented method comprising:
under the control of a computer system comprising computer hardware, the computer system configured with computer executable instructions:
obtaining input media depicting a real person as captured by a camera;
obtaining base three-dimensional (3D) model data associated with a generic human figure;
providing at least a portion of the input media to a machine learning model configured to extract visual information regarding one or more humans depicted in image or video data;
modifying the base 3D model data based on visual information extracted by the machine learning model to generate custom 3D model data corresponding to the real person;
extracting, from the input media, visual information regarding a first item worn by the real person as depicted in the input media, wherein the first item comprises a clothing item or an accessory;
generating or selecting a virtual item corresponding to the first item worn by the real person; and
storing, in a data store that stores virtual character data for virtual characters that populate a virtual world in at least one video game, data defining a custom virtual character, wherein the data defining the custom virtual character comprises at least the custom 3D, wherein the data store further includes an association between the virtual item and the custom virtual character.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, further comprising rendering, within a 3D virtual environment of a video game, a series of frames for display that depict the custom virtual character performing one or more actions, wherein a visual appearance of the custom virtual character as rendered is based at least in part on the custom 3D model data and includes depiction of the virtual character wearing the virtual item.

### Claim at position 12

12-16. (canceled)

### Claim 17

A non-transitory computer-readable storage medium having stored thereon computer-readable instructions that, when executed, configure a computing system to:
obtain input media depicting a real person as captured by a camera;
obtain base three-dimensional (3D) model data associated with a generic human figure;
provide at least a portion of the input media to a machine learning model configured to extract visual information regarding one or more humans depicted in image or video data;
modify the base 3D model data based on visual information extracted by the machine learning model to generate custom 3D model data corresponding to the real person;
extract, from the input media, visual information regarding a first item worn by the real person as depicted in the input media, wherein the first item comprises a clothing item or an accessory;
generate or select a virtual item corresponding to the first item worn by the real person; and
store, in a data store that stores virtual character data for virtual characters that populate a virtual world in at least one video game, data defining a custom virtual character, wherein the data defining the custom virtual character comprises at least the custom 3D model, wherein the data store further includes an association between the virtual item and the custom virtual character.

### Claim 18 (depends on 17)

The non-transitory computer-readable storage medium of claim 17, wherein the instructions further configure the computing system to render, within a 3D virtual environment of a video game, a series of frames for display that depict the custom virtual character performing one or more actions, wherein a visual appearance of the custom virtual character as rendered is based at least in part on the custom 3D model data and includes depiction of the virtual character wearing the virtual item.

### Claim 19 (depends on 17)

The non-transitory computer-readable storage medium of claim 17, wherein the real person is a player of the at least one video game, and wherein the input media is obtained from at least one of (a) a computing device of the player or (b) a third party media account associated with the player.

### Claim 20

(canceled)

### Claim 21 (depends on 10)

The computer-implemented method of claim 10, wherein the machine learning model applies an image-to-image translation using a conditional generative adversarial network (cGAN).

### Claim 22 (depends on 10)

The computer-implemented method of claim 10, wherein modifying the base 3D model data based on the visual information extracted by the machine learning model includes:
applying a gradient descent technique comprising repeatedly adjusting a position and pose of a virtual camera until a rendered 3D virtual character based on the base 3D model data is positioned and posed similarly to a position and pose of the real person within a particular image or video frame of the input media; and
adjusting one or more model parameters of the base 3D model data such that a subsequently rendered 3D virtual character based on adjusted base 3D model data appears similar to the real person depicted in the particular image or video frame of the input media.

### Claim 23 (depends on 10)

The computer-implemented method of claim 10, wherein the extracting the visual information regarding the first item worn by the real person comprises applying a classifier trained to classify one or more objects detected in an image as one of a plurality of predefined types of clothing items.

### Claim 24 (depends on 23)

The computer-implemented method of claim 23, wherein each of the predefined types of clothing items are associated with (a) corresponding 3D model data of a generic clothing item in an electronic data store and (b) a visual style extraction model trained to customize at least one of a texture or 3D mesh of the corresponding 3D model data based on the input media.

### Claim 25 (depends on 23)

The computer-implemented method of claim 23, wherein the plurality of predefined types of clothing items comprise two or more of a shirt, a jacket, a hat, short pants, long pants, boots or shoes.

### Claim 26 (depends on 10)

The computer-implemented method of claim 10, wherein the extracting the visual information regarding the first item worn by the real person comprises applying a classifier trained to classify one or more objects detected in an image as one of a plurality of predefined types of accessories.

### Claim 27 (depends on 26)

The computer-implemented method of claim 26, wherein the plurality of predefined types of accessories comprise two or more of glasses, a belt, gloves, earrings, a necklace or a watch.