# US 2021/0279844 A1 · Method to provide a video with a computer-modified visual of a desired face of a person

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279844 A1
- Document ID: US20210279844A1
- Status: Published application
- Kind: A1
- Application number: 17/331,195
- Filed: 2021-05-26
- Published: 2021-09-09
- Earliest claimed: 2019-12-11
- Assignees: Quantiface GmbH
- Applicants: QuantiFace GmbH
- Inventors: Heike BENDITTE-KLEPETKO
- Web page: https://patentgrep.com/patent/US20210279844A1

## CPC

- A61B 5/167: {Personality evaluation}
- A61B 5/004: {adapted for image acquisition of a particular organ or body part (A61B5/0082 takes precedence; arrangements for optical scanning A61B5/0062)}
- A61B 5/7267: {involving training the classification device}
- G06Q 10/10: Office automation; Time management
- G06Q 10/40: Business processes related to social networking or social networking services
- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06T 7/11: Region-based segmentation
- G06T 7/194: involving foreground-background segmentation
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06V 40/174: {Facial expression recognition}
- G11B 27/036: Insert-editing
- H04N 7/15: Conference systems
- A61B 5/0037: {Performing a preliminary scan, e.g. a prescan for identifying a region of interest}
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/30201: Face
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

At least one characteristic of a face of the person to be improved is inputted. Artificial intelligence is used to analyse a visual of the person's face and generate data sets of modifications to improve the visual appearance of the person in different ways towards a selected characteristic. The visual of the face of the person is modified based on the data sets of modifications and computer-modified visuals of the face of the person are generated and displayed. One of the computer-modified visuals is selected as the desired face of the person. A video is provided that shows a computer-modified visual of the desired face of the person.

## Claims

### Claim 1

A method for providing a video with a computer-modified visual of a desired face of a person, wherein the method comprises the steps of
A. generating a data set of visuals of faces and extracted face property data thereof linked to face characteristics data provided by a representative set of humans that rate the visuals of these faces about their face characteristics and store the data set in a database;
B. extracting further face property data of these visuals of faces and use these extracted face property data together with the generated data set for training of an artificial intelligence to enable the artificial intelligence to provide an automated rating of the characteristics of the visuals of faces;
C. providing at least one visual of the face of the person;
D. inputting at least one characteristic of the face of the person to be improved;
E. using the artificial intelligence to analyse the visual of the person's face and to generate at least two data sets of modifications to improve the visual appearance of the person in at least two different ways towards the selected characteristic;
F. modifying the visual of the face of the person based on both data sets of modifications and generate two computer-modified visuals of the face of the person;
G. displaying both computer-modified visuals of the face of the person;
H. selecting one of the two computer-modified visuals as the desired face of the person; and
I. providing a video that shows the face of the person and use the selected data set of modifications of the selected desired face of the person to modify the video to provide the video with the computer-modified visual of the desired face of the person.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the method comprises the following steps for above method step I:
continuously:
identifying the visual of the face of the person in the video;
modifying the visual of the identified face of the person based on the selected data set of modifications and generate the computer-modified visual of the desired face of the person;
replacing the identified visual of the face of the person with the computer-modified visual of the desired face of the person in the video; and
displaying the video with the replaced computer-modified visual of the desired face of the person.

### Claim 3 (depends on 1)

The method according to claim 1, wherein method steps A and B are processed to prepare a computer to process above method step E and that above method steps C to H are processed to select the desired face of the person and to process step I for the modification of a live video in particular of a video conference.

### Claim 4 (depends on 1)

The method according to claim 1, wherein method step E generates only data sets of modifications to improve the visual appearance of the person within the borders of visual modifications of a face achievable by cosmetic and/or medical treatments and/or manual changes of the visual appearance of the person.

### Claim 5 (depends on 4)

The method according to claim 4, wherein method step B furthermore comprises the following step:
generating a data set of modifications of a face achievable by cosmetic and/or medical treatments and/or manual changes of the visual appearance of the person and store the data set in the database.

### Claim 6 (depends on 5)

The method according to claim 5, wherein the method comprises the following further step of:
using predefined samples of tools, like in particular glasses or earrings, to generate the data set of modifications of a face achievable by manual changes of the visual appearance of the person.

### Claim 7 (depends on 5)

The method according to claim 5, wherein the method comprises the following further step of:
providing at least one visual of a personal object of the person, like in particular glasses or earrings, to generate the data set of modifications of a face achievable by manual changes of the visual appearance of the person.

### Claim 8 (depends on 1)

The method according to claim 1, wherein method steps C to H are processed for two or more characteristics or combinations of characteristics and data sets of modifications selected for these different characteristics are stored and that the artificial intelligence is built to analyse the actual facial expressions of the face of the person in the video and to recognize content and/or tone of the person and to select the appropriate stored data set of modifications that supports the content and/or tone of the person and to dynamically modify the visual of the face of the person in the video with the selected data set of modifications.

### Claim 9 (depends on 1)

Method The method according to claim 1, characterized in that wherein the method comprises the following further step of:
pre-processing the provided visual of the face of the person to standardize their content prior to the analyse of the artificial intelligence, wherein pre-processing may include one or more of the following steps: cut-out the background behind the face from the visual; cut-out the ears of the person's face to reduce their visual influence; cut-out clothes and other wardrobe that might influence with the face; cut-out the head hair of the person.

### Claim 10 (depends on 1)

A device for providing a video with a computer-modified visual of a desired face of a person, which device comprises a display and input means to input at least one characteristic of the face of a person to be changed and to provide at least one visual of the face of the person, wherein:
the device comprises a deep learning based application to process the steps A and B of the method according to claim 1 to generate a database with data sets of visuals of faces and extracted face property data thereof linked to face characteristics and,
that the deep learning based application is built to process the step E of the method according to claim 1 to analyse the provided visual of the person's face and to generate at least two data sets of modifications to improve the visual appearance of the person in at least two different ways towards the selected characteristic and,
that modification means are built to process the step [[6]] F of the method according to claim 1 to modify the visual of the face of the person based on both data sets of modifications and to generate two computer-modified visuals of the face of the person and, that the device is built to display both computer-modified visuals of the face of the person and based on the persons' selection of the desired face with the input means the modification means are built to modify a video that shows the face of the person by the use of the selected data set of modifications of the selected desired face of the person to modify the video to provide a video with a computer-modified visual of a desired face of a person.

### Claim 11 (depends on 1)

A computer-program comprising instructions which, when the program is executed by a device, causes the device to carry out the steps of a method claimed in claim 1 to provide a video with a computer-modified visual of a desired face of a person.