# US RE49,044 E1 · Automatic avatar creation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US RE49,044 E1
- Document ID: USRE49044E1
- Status: Granted
- Kind: E1
- Application number: 16/147,348
- Filed: 2018-09-28
- Granted: 2022-04-19
- Earliest claimed: 2010-06-01
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Alex Nelson; Cedric Bray; Thomas Goossens; Rudolph Van Der Merwe; Richard E. Crandall; Bertrand Serlet
- Web page: https://patentgrep.com/patent/USRE49044E1

## CPC

- G06K 9/00281
- A63F 2300/5553: user representation in the game field, e.g. avatar
- G06K 9/00221
- G06K 9/00268
- G06K 9/00315
- H04N 21/44218: {Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45)}
- G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A three-dimensional (“3D”) avatar can be automatically created that resembles the physical appearance of an individual captured in one or more input images or video frames. The avatar can be further customized by the individual in an editing environment and used in various applications, including but not limited to gaming, social networking and video conferencing.

## Claims

### Claim 1

A computer implemented method, comprising:
receiving one or more digital images of an individual;
determining, in a computer, digital representations of facial elements of the individual from the one or more digital images;
searching a population of avatars using a genetic process, where the genetic process includes using the computer to evaluate the overall fitness of the avatars using a fitness function that compares the digital representations of the individual's facial elements to digital representations of corresponding avatar facial elements; and
selecting or receiving a selection of one or more avatars from the population of avatars based on fitness values associated with the avatars;
wherein evaluating the overall fitness of the avatars using a fitness function includes calculating fitness values for individual facial elements of the avatars based on linear and geometric weights corresponding to each of the individual facial elements.

### Claim 2 (depends on 1)

The method of claim 1, where determining digital representations of facial elements of the individual, further comprises:
image processing the input images to identify the individual's facial elements; and
generating binary masks or images for the identified facial elements.

### Claim 3 (depends on 1)

The method of claim 1, where the digital representations of the corresponding avatar facial elements are binary masks.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
initializing the genetic process by randomly generating a population of avatars having the facial elements.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
initializing the genetic process by seeding the avatar population with avatars having the facial elements.

### Claim 6 (depends on 1)

The method of claim 1, where evaluating the overall fitness of the avatars using a fitness function, further comprises: calculating the overall fitness values for the avatars based on the individual fitness values.

### Claim 7 (depends on 6)

The method of claim 6, where the overall fitness function is given by
f
=
∑
i
⁢
α
i
·
f
i
γ
i
,
where αi and γi are linear and geometric weights for each facial element fitness value fiγi, respectively, and i is an index into the facial elements.

### Claim 8 (depends on 1)

The method of claim 1, where the facial elements are from a group of facial elements including face, eyebrows, eyes, nose, mouth, chin and hair.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
preprocessing the input images using one or more of the following processes: resizing, color space conversion, inverted square root y channel, fine texture, cartoon texture and face detection.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
presenting the selected one or more avatars on a touch sensitive display; and
receiving input from the touch sensitive display selecting an avatar.

### Claim 11

A system, comprising:
a processor; and
a computer-readable storage medium coupled to the processor and including instructions, which, when executed by the processor, causes the processor to perform operations comprising:
receiving one or more digital images of an individual;
determining, in a computer, digital representations of facial elements of the individual from the one or more digital images;
searching a population of avatars using a genetic process, where the genetic process includes using the computer to evaluate the overall fitness of the avatars using a fitness function that compares the digital representations of the individual's facial elements to digital representations of corresponding avatar facial elements; and
selecting or receiving a selection of one or more avatars from the population of avatars based on fitness values associated with the avatars;
wherein evaluating the overall fitness of the avatars using a fitness function includes calculating fitness values for individual facial elements of the avatars based on linear and geometric weights corresponding to each of the individual facial elements.

### Claim 12 (depends on 11)

The system of claim 11, where determining digital representations of facial elements of the individual, further comprises:
image processing the input images to identify the individual's facial elements; and
generating binary masks or images for the identified facial elements.

### Claim 13 (depends on 11)

The system of claim 11, where the digital representations of the corresponding avatar facial elements are binary masks.

### Claim 14 (depends on 11)

The system of claim 11, where evaluating the overall fitness of the avatars using a fitness function, further comprises: calculating the overall fitness values for the avatars based on the individual fitness values.

### Claim 15 (depends on 11)

The system of claim 11, further comprising:
preprocessing the input images using one or more of the following processes: resizing, color space conversion, inverted square root y channel, fine texture, cartoon texture and face detection.

### Claim 16 (depends on 11)

The system of claim 11, further comprising:
presenting the selected one or more avatars on a touch sensitive display; and
receiving input from the touch sensitive display selecting an avatar.

### Claim 17

A non-transitory computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, causes the one or more processors to perform operations comprising:
receiving one or more digital images of an individual;
determining, in a computer, digital representations of facial elements of the individual from the one or more digital images;
searching a population of avatars using a genetic process, where the genetic process includes using the computer to evaluate the overall fitness of the avatars using a fitness function that compares the digital representations of the individual's facial elements to digital representations of corresponding avatar facial elements; and
selecting or receiving a selection of one or more avatars from the population of avatars based on fitness values associated with the avatars;
wherein evaluating the overall fitness of the avatars using a fitness function includes calculating fitness values for individual facial elements of the avatars based on linear and geometric weights corresponding to each of the individual facial elements.

### Claim 18 (depends on 17)

The non-transitory computer-readable storage medium of claim 17, where determining digital representations of facial elements of the individual, further comprises:
image processing the input images to identify the individual's facial elements; and
generating binary masks or images for the identified facial elements.

### Claim 19 (depends on 17)

The non-transitory computer-readable storage medium of claim 17, where the digital representations of the corresponding avatar facial elements are binary masks.

### Claim 20 (depends on 17)

The non-transitory computer-readable storage medium of claim 17, where evaluating the overall fitness of the avatars using a fitness function, further comprises: calculating the overall fitness values for the avatars based on the individual fitness values.

### Claim 21

A method, comprising:
at an electronic device with a display, a camera, and one or more input devices:
displaying, on the display, a first user interface that provides information about generation, by the electronic device, of an avatar of a user of the electronic device;
while displaying the first user interface, receiving, via the one or more input devices, a request to initiate the generation of the avatar based on a digital representation of the user of the electronic device that is being captured by the camera of the electronic device;
in response to receiving the request to initiate the generation of the avatar, while capturing the digital representation of the user, displaying, on the display, a second user interface that includes:
the digital representation of the user as it is being captured by the camera of the electronic device, and
a visual indication of a current status of evaluating the digital representation of the user of the electronic device that is being captured by the camera of the electronic device, wherein:
the visual indication is overlaid on the digital representation of the user as the digital representation of the user is being evaluated, and
the visual indication moves from being overlaid on a first portion of the digital representation of the user corresponding to a first facial element to being overlaid on a second portion, different from the first portion, of the digital representation of the user corresponding to a second facial element, different from the first facial element, as the digital representation of the user is being evaluated;
after displaying the second user interface, displaying, on the display, a first proposed avatar based on the digital representation of the user; and
while displaying the first proposed avatar, providing the user with options for customizing the first proposed avatar.

### Claim 22 (depends on 21)

The method of claim 21, further comprising:
after displaying the second user interface, concurrently displaying, on the display, the first proposed avatar and a second proposed avatar based on the digital representation of the user.

### Claim 23 (depends on 22)

The method of claim 22, further comprising:
while concurrently displaying the first proposed avatar and the second proposed avatar on the display, receiving, via the one or more input devices, an input corresponding to a selection of a respective proposed avatar of the first and second proposed avatars; and
in response to receiving the input, entering a manual editing mode on the electronic device for manually editing the respective proposed avatar.

### Claim 24 (depends on 21)

The method of claim 21, wherein displaying the second user interface includes displaying dynamic feedback for guiding a position of the user that will improve an accuracy of the avatar that is being generated by the electronic device, and the feedback for guiding the position of the user comprises feedback for constraining the position of the user.

### Claim 25 (depends on 24)

The method of claim 24, wherein the feedback for constraining the position of the user comprises feedback for constraining a position of the user's head while the camera captures successive images of the user.

### Claim 26 (depends on 21)

The method of claim 21, further comprising:
while displaying the first proposed avatar on the display, receiving, via the one or more input devices, a touch input from the user with respect to the first proposed avatar; and
in response to receiving the touch input, performing a corresponding action on the electronic device with respect to the first proposed avatar.

### Claim 27 (depends on 26)

The method of claim 26, wherein the touch input with respect to the first proposed avatar comprises touching of the first proposed avatar on the display.

### Claim 28 (depends on 26)

The method of claim 26, wherein the corresponding action comprises changing a display characteristic of the first proposed avatar on the display.

### Claim 29 (depends on 28)

The method of claim 28, wherein changing the display characteristic of the first proposed avatar on the display comprises enlarging the first proposed avatar on the display.

### Claim 30 (depends on 21)

The method of claim 21, wherein the visual indication of the current status indicates a respective portion of the user currently being evaluated.

### Claim 31 (depends on 21)

The method of claim 21, wherein the second user interface includes a latest candidate avatar of the user, overlaid on the digital representation of the user as it is being captured by the camera, based on a current progress of the evaluation of the digital representation of the user.

### Claim 32

A non-transitory computer-readable medium storing instructions, which when executed by one or more processors of an electronic device with a display, a camera, and one or more input devices, cause the one or more processors to perform a method comprising:
displaying, on the display, a first user interface that provides information about generation, by the electronic device, of an avatar of a user of the electronic device;
while displaying the first user interface, receiving, via the one or more input devices, a request to initiate the generation of the avatar based on a digital representation of the user of the electronic device that is being captured by the camera of the electronic device;
in response to receiving the request to initiate the generation of the avatar, while capturing the digital representation of the user, displaying, on the display, a second user interface that includes:
the digital representation of the user as it is being captured by the camera of the electronic device, and
a visual indication of a current status of evaluating the digital representation of the user of the electronic device that is being captured by the camera of the electronic device, wherein:
the visual indication is overlaid on the digital representation of the user as the digital representation of the user is being evaluated, and
the visual indication moves from being overlaid on a first portion of the digital representation of the user corresponding to a first facial element to being overlaid on a second portion, different from the first portion, of the digital representation of the user corresponding to a second facial element, different from the first facial element, as the digital representation of the user is being evaluated;
after displaying the second user interface, displaying, on the display, a first proposed avatar based on the digital representation of the user; and
while displaying the first proposed avatar, providing the user with options for customizing the first proposed avatar.

### Claim 33 (depends on 32)

The non-transitory computer-readable medium of claim 32, the method further comprising:
after displaying the second user interface, concurrently displaying, on the display, the first proposed avatar and a second proposed avatar based on the digital representation of the user.

### Claim 34 (depends on 33)

The non-transitory computer-readable medium of claim 33, the method further comprising:
while concurrently displaying the first proposed avatar and the second proposed avatar on the display, receiving, via the one or more input devices, an input corresponding to a selection of a respective proposed avatar of the first and second proposed avatars; and
in response to receiving the input, entering a manual editing mode on the electronic device for manually editing the respective proposed avatar.

### Claim 35 (depends on 32)

The non-transitory computer-readable medium of claim 32, wherein displaying the second user interface includes displaying dynamic feedback for guiding a position of the user that will improve an accuracy of the avatar that is being generated by the electronic device, and the feedback for guiding the position of the user comprises feedback for constraining the position of the user.

### Claim 36 (depends on 35)

The non-transitory computer-readable medium of claim 35, wherein the feedback for constraining the position of the user comprises feedback for constraining a position of the user's head while the camera captures successive images of the user.

### Claim 37 (depends on 32)

The non-transitory computer-readable medium of claim 32, the method further comprising:
while displaying the first proposed avatar on the display, receiving, via the one or more input devices, a touch input from the user with respect to the first proposed avatar; and
in response to receiving the touch input, performing a corresponding action on the electronic device with respect to the first proposed avatar.

### Claim 38 (depends on 37)

The non-transitory computer-readable medium of claim 37, wherein the touch input with respect to the first proposed avatar comprises touching of the first proposed avatar on the display.

### Claim 39 (depends on 37)

The non-transitory computer-readable medium of claim 37, wherein the corresponding action comprises changing a display characteristic of the first proposed avatar on the display.

### Claim 40 (depends on 39)

The non-transitory computer-readable medium of claim 39, wherein changing the display characteristic of the first proposed avatar on the display comprises enlarging the first proposed avatar on the display.

### Claim 41 (depends on 32)

The non-transitory computer-readable medium of claim 32, wherein the visual indication of the current status indicates a respective portion of the user currently being evaluated.

### Claim 42 (depends on 32)

The non-transitory computer-readable medium of claim 32, wherein the second user interface includes a latest candidate avatar of the user, overlaid on the digital representation of the user as it is being captured by the camera, based on a current progress of the evaluation of the digital representation of the user.

### Claim 43

An electronic device comprising:
one or more processors;
a display;
a camera;
one or more input devices; and
memory storing instructions, which when executed by the one or more processors, cause the one or more processors to perform a method comprising:
displaying, on the display, a first user interface that provides information about generation, by the electronic device, of an avatar of a user of the electronic device;
while displaying the first user interface, receiving, via the one or more input devices, a request to initiate the generation of the avatar based on a digital representation of the user of the electronic device that is being captured by the camera of the electronic device;
in response to receiving the request to initiate the generation of the avatar, while capturing the digital representation of the user, displaying, on the display, a second user interface that includes:
the digital representation of the user as it is being captured by the camera of the electronic device, and
a visual indication of a current status of evaluating the digital representation of the user of the electronic device that is being captured by the camera of the electronic device, wherein:
the visual indication is overlaid on the digital representation of the user as the digital representation of the user is being evaluated, and
the visual indication moves from being overlaid on a first portion of the digital representation of the user corresponding to a first facial element to being overlaid on a second portion, different from the first portion, of the digital representation of the user corresponding to a second facial element, different from the first facial element, as the digital representation of the user is being evaluated;
after displaying the second user interface, displaying, on the display, a first proposed avatar based on the digital representation of the user; and
while displaying the first proposed avatar, providing the user with options for customizing the first proposed avatar.

### Claim 44 (depends on 43)

The electronic device of claim 43, the method further comprising:
after displaying the second user interface, concurrently displaying, on the display, the first proposed avatar and a second proposed avatar based on the digital representation of the user.

### Claim 45 (depends on 44)

The electronic device of claim 44, the method further comprising:
while concurrently displaying the first proposed avatar and the second proposed avatar on the display, receiving, via the one or more input devices, an input corresponding to a selection of a respective proposed avatar of the first and second proposed avatars; and
in response to receiving the input, entering a manual editing mode on the electronic device for manually editing the respective proposed avatar.

### Claim 46 (depends on 43)

The electronic device of claim 43, wherein displaying the second user interface includes displaying dynamic feedback for guiding a position of the user that will improve an accuracy of the avatar that is being generated by the electronic device, and the feedback for guiding the position of the user comprises feedback for constraining the position of the user.

### Claim 47 (depends on 46)

The electronic device of claim 46, wherein the feedback for constraining the position of the user comprises feedback for constraining a position of the user's head while the camera captures successive images of the user.

### Claim 48 (depends on 43)

The electronic device of claim 43, the method further comprising:
while displaying the first proposed avatar on the display, receiving, via the one or more input devices, a touch input from the user with respect to the first proposed avatar; and
in response to receiving the touch input, performing a corresponding action on the electronic device with respect to the first proposed avatar.

### Claim 49 (depends on 48)

The electronic device of claim 48, wherein the touch input with respect to the first proposed avatar comprises touching of the first proposed avatar on the display.

### Claim 50 (depends on 48)

The electronic device of claim 48, wherein the corresponding action comprises changing a display characteristic of the first proposed avatar on the display.

### Claim 51 (depends on 50)

The electronic device of claim 50, wherein changing the display characteristic of the first proposed avatar on the display comprises enlarging the first proposed avatar on the display.

### Claim 52 (depends on 43)

The electronic device of claim 43, wherein the visual indication of the current status indicates a respective portion of the user currently being evaluated.

### Claim 53 (depends on 43)

The electronic device of claim 43, wherein the second user interface includes a latest candidate avatar of the user, overlaid on the digital representation of the user as it is being captured by the camera, based on a current progress of the evaluation of the digital representation of the user.