# US 2019/0171869 A1 · Creating videos with facial expressions

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0171869 A1
- Document ID: US20190171869A1
- Status: Published application
- Kind: A1
- Application number: 16/320,966
- Filed: 2017-07-25
- Published: 2019-06-06
- Earliest claimed: 2016-07-25
- Assignees: Bgr Technologies Pty Limited
- Applicants: Bgr Technologies Pty Limited
- Inventors: Ron FORTUNE; Bill BAILEY
- Web page: https://patentgrep.com/patent/US20190171869A1

## CPC

- G06V 40/176: {Dynamic expression}
- G06T 7/00: Image analysis
- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06T 13/80: Two-dimensional [2D] animation, e.g. using sprites
- G06V 40/168: {Feature extraction; Face representation}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06V 40/175: {Static expression}
- G06T 2210/44: Morphing
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The present disclosure relates to creating videos. A mobile device creates a graphic user interface to capture by the camera of the device multiple photographic facial images of a user for respective multiple facial expressions of a character in the video. Using the multiple photographic facial images, the device modifies stored character images by matching facial features of the character to facial features of the user for the multiple facial expressions of the character in the video and creates the video based on the modified character images. The facial expression of the user is used to influence the facial expression of the character. This method enables replacement of certain visual style elements with a given user's own style elements.

## Claims

### Claim at position 1

-13. (canceled)

### Claim 14

A method for creating an output frame for a character in a video, the method comprising:
determining an estimated reference facial feature of the character based on a first reference facial image and a second reference facial image of the character;
determining an estimated user facial feature of a user based on a first photographic facial image and a second photographic facial image of the user;
determining a transformation that transforms the estimated reference facial feature of the character into an approximation or representation of the estimated user facial feature of the user;
modifying, based on the transformation, a third reference facial image of the character associated with the estimated reference facial feature of the character; and
creating the output frame for the character in the video based on the modified third reference facial image.

### Claim 15 (depends on 14)

The method of claim 14, wherein determining the estimated reference facial feature of the character comprises:
determining a first distance between a first reference facial feature of the first reference facial image of the character and a second reference facial feature of the second reference facial image of the character; and
determining the estimated reference facial feature of the character based on the first distance, the first reference facial feature and the second reference facial feature.

### Claim 16 (depends on 15)

The method of claim 15, wherein determining the estimated reference facial feature of the character comprises performing an interpolation operation based on the first reference facial feature and the second reference facial feature with respect to the first distance.

### Claim 17 (depends on 15)

The method of claim 15, wherein determining the estimated reference facial feature of the character comprises performing an extrapolation operation based on the first reference facial feature and the second reference facial feature with respect to the first distance.

### Claim 18 (depends on 15)

The method of claim 15, wherein the first reference facial feature includes a first set of control points, and the second reference facial feature includes a second set of control points, and the first distance is indicative of a distance between the first set of control points and the second set of control points.

### Claim 19 (depends on 15)

The method of claim 15, wherein determining the estimated user facial feature of the user comprises:
determining a second distance between a user first facial feature of the first photographic facial image of the user and a user second facial feature of the second photographic facial image of the user; and
determining the estimated user facial feature based on the second distance, the user first facial feature and the user second facial feature.

### Claim 20 (depends on 19)

The method of claim 19, wherein determining the estimated user facial feature of the user comprises performing an interpolation operation based on the user first facial feature and the user second facial feature with respect to the second distance.

### Claim 21 (depends on 19)

The method of claim 19, wherein determining the estimated user facial feature of the user comprises performing an extrapolation operation based on the user first facial feature and the user second facial feature with respect to the second distance.

### Claim 22 (depends on 19)

The method of any one of claims 19, wherein the user first facial feature includes a third set of control points, and the user second facial feature includes a fourth set of control points, and the second distance is indicative of a distance between the third set of control points and the fourth set of control points.

### Claim 23 (depends on 22)

The method of claim 22, wherein modifying the third reference facial image of the character comprises transforming a first spline curve represented by the estimated reference facial feature of the character into an approximation or representation of a second spline curve represented by the estimated user facial feature of the user.

### Claim 24 (depends on 1)

A computer software product, including machine-readable instructions, when executed by a processor of a mobile device, that causes the processor to perform the method of claim 1.

### Claim 25

A mobile device for creating an output frame for a character in a video, the mobile device comprising
a camera to capture a first photographic facial image and a second photographic facial image of the user;
a display; and
a processor, the processor configured to
determine an estimated reference facial feature of the character based on a first reference facial image and a second reference facial image of the character;
determine an estimated user facial feature of the user based on the first photographic facial image and the second photographic facial image of the user;
determine a transformation that transforms the estimated reference facial feature of the character into an approximation or representation of the estimated user facial feature of the user;
modify, based on the transformation, a third reference facial image of the character associated with the estimated reference facial feature of the character;
create the output frame based on the modified third reference facial image; and
present the output frame on the display.
