# US 2019/0378316 A1 · Dynamic real-time generation of three-dimensional avatar models of users based on live visual input of users' appearance and computer systems and computer-implemented methods directed to thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0378316 A1
- Document ID: US20190378316A1
- Status: Published application
- Kind: A1
- Application number: 16/547,436
- Filed: 2019-08-21
- Published: 2019-12-12
- Earliest claimed: 2017-04-18
- Assignees: Banuba Limited
- Applicants: Banuba Limited
- Inventors: Yury Hushchyn; Dzmitry Kachatkou
- Web page: https://patentgrep.com/patent/US20190378316A1

## CPC

- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06T 17/30: Polynomial surface description
- G06V 40/165: {using facial parts and geometric relationships}
- G06V 40/167: {using comparisons between temporally consecutive images}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06V 40/174: {Facial expression recognition}
- H04L 65/762: {at the source (reformatting of additional data in video distribution servers H04N21/2355)}
- H04N 7/157: {defining a virtual conference space and using avatars or agents (computer conference optimisation or adaptation H04L12/1827)}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

In some embodiments, the present invention provides for a computer system that may include a camera component configured to acquire a visual content, where the visual content includes a plurality of frames having a visual representation of a person's face; and a processor configured to: train a face detection regressor with a synthetic face model database to obtain a face detection trained regressor; apply, for each frame, the face detection trained regressor to detect or to track the face based on facial features, local features, and a pre-defined hyperparameter; construct an intermediate multi-dimensional face model; apply machine learning to determine features of an intermediate multi-dimensional head model; construct a multi-dimensional avatar; and utilize the multi-dimensional avatar to perform an activity associated with the person.

## Claims

### Claim 1

A method, comprising:
training, by at least one processor, at least one face detection regressor with at least one synthetic face model database to obtain at least one face detection trained regressor;
wherein the at least one synthetic face model database comprises a plurality of facial features;
wherein each facial feature is identified by a pre-defined set of parameters;
obtaining, by the at least one processor, a plurality of frames having a visual representation of a face of at least one person;
extracting, by the at least one processor, a plurality of local features from each frame of the plurality of frames;
applying, by the at least one processor, for each frame, the at least one face detection trained regressor to the plurality of local features to detect or to track a presence of a face of a particular individual, based, at least in part, on:
i) the plurality of facial features,
ii) the plurality of local features, and
iii) at least one pre-defined hyperparameter;
constructing, by the at least one processor, an intermediate multi-dimensional face model of the face of the particular individual based on the detected or tracked presence of the face of the particular individual;
applying, by the at least one processor, machine learning to determine a plurality of features of an intermediate multi-dimensional head model of the particular individual based on the detected or tracked presence of the face of the particular individual;
constructing, by the at least one processor, at least one multi-dimensional avatar of the particular individual based, at least in part, on:
i) the intermediate multi-dimensional face model of the particular individual, and
ii) the plurality of features of the intermediate multi-dimensional head model of the particular individual; and
utilizing, by the at least one processor, the at least one multi-dimensional avatar of the particular individual to perform at least one activity associated with the particular individual.

### Claim 2

A system comprising:
a camera component, wherein the camera component is configured to acquire a visual content, wherein the visual content comprises a plurality of frames having a visual representation of a face of at least one person; and
at least one processor configured to:
train at least one face detection regressor with at least one synthetic face model database to obtain at least one face detection trained regressor;
wherein the at least one synthetic face model database comprises a plurality of facial features;
wherein each facial feature is identified by a pre-defined set of parameters;
extract a plurality of local features from each frame of the plurality of frames;
apply, for each frame, the at least one face detection trained regressor to the plurality of local features to detect or to track a presence of a face of a particular individual, based, at least in part, on:
i) the plurality of facial features,
ii) the plurality of local features, and
iii) at least one pre-defined hyperparameter;
construct an intermediate multi-dimensional face model of the face of the particular individual based on the detected or tracked presence of the face of the particular individual;
apply machine learning to determine a plurality of features of an intermediate multi-dimensional head model of the particular individual based on the detected or tracked presence of the face of the particular individual;
construct at least one multi-dimensional avatar of the particular individual based, at least in part, on:
i) the intermediate multi-dimensional face model of the particular individual, and
ii) the plurality of features of the intermediate multi-dimensional head model of the particular individual; and
utilize the at least one multi-dimensional avatar of the particular individual to perform at least one activity associated with the particular individual.
