# US 2024/0177283 A1 · System and method for an audio-visual avatar evaluation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0177283 A1
- Document ID: US20240177283A1
- Status: Published application
- Kind: A1
- Application number: 18/059,395
- Filed: 2022-11-28
- Published: 2024-05-30
- Earliest claimed: 2022-11-28
- Assignees: Alemira AG; Constructor Education and Research Genossenschaft
- Applicants: Alemira AG; Constructor Education and Research Genossenschaft
- Inventors: Ilya Baimetov; Denis Parkhomenko; Marcel de Korte; Ivan Kirillov; Dmitriy Obukhov; Alexey Rybak; Laurent Dedenis; Serg Bell; Stanislav Protasov
- Web page: https://patentgrep.com/patent/US20240177283A1

## CPC

- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 25/57: for processing of video signals
- G10L 25/60: for measuring the quality of voice signals
- G10L 25/84: for discriminating voice from noise
- G10L 25/90: Pitch determination of speech signals
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/30196: Human being; Person
- G10L 15/01: Assessment or evaluation of speech recognition systems
- G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

The present disclosure relates to a system to evaluate an avatar generated by an avatar generator. The system comprises an evaluation module including an audio evaluation module for evaluating audio features and a video evaluation module for evaluating video features. Evaluation of the avatar includes extracting audio and video features from the avatar and applying a set of evaluation metrics for generating audio and video evaluation scores. The scores are combined to generate a final score. For avatar generator evaluation, audio clip and video clip are provided to the audio evaluation module and video evaluation module, respectively. A set of evaluation metrics is applied for evaluation. Each metric can generate a score. All scores are combined to generate a final evaluation score.

## Claims

### Claim 1

A method for automated evaluation of an avatar generated by an avatar generator comprising the steps of:
obtaining, by an audio evaluator, a speech generated by a TTS module;
obtaining, by a video evaluator, a video clip generated by a video generator;
obtaining, by the audio evaluator, the audio features of the target person;
obtaining, by the video evaluator, the video features of the target person;
comparing the speech with the audio features of the target person using a set of audio metrics, and generating an audio evaluation score for the speech;
comparing the video clip with the video features of the target person using a set of video metrics, and generating a video evaluation score for the video clip;
combining the audio evaluation score and the video evaluation score; and
generating a combined naturalness score for the avatar generator based on the combined score of the audio evaluation score and the video evaluation score.

### Claim 2 (depends on 1)

The method of claim 1, wherein the step of evaluating the speech further comprises:
a. evaluating speech intelligibility using ASR based evaluation metrics;
b. evaluating audio noise level using VAD based evaluation metrics;
c. evaluating naturalness of speech intonation using pitch-based metrics;
d. evaluating voice similarities using EER and COS metrics; and
e. evaluating speech pronunciation statistics.

### Claim 3 (depends on 2)

The method of claim 2, wherein evaluation scores generated by each of the set of audio metrics are combined to generate the audio evaluation score.

### Claim 4 (depends on 1)

The method of claim 1, wherein the step of evaluating the video clip further comprises:
a. evaluating a video quality with a reference image using PSNR, MS-SSIM, FSIM, LPIPS, VMAF, VIF, NLP metrics;
b. evaluating a video quality with no reference images using WaDIQaM, DUBCNN, TRES, ChipQA;
c. evaluating distribution using distribution-based metrics;
d. evaluating lip synchronization using lip synchronization metrics; and
e. evaluating an identity of the target using identity metrics.

### Claim 5 (depends on 4)

The method of claim 4, wherein the evaluation scores generated by each of the set of video metrics are combined to generate a video evaluation score.

### Claim 6 (depends on 1)

The method of claim 1, the step of generating a combined naturalness score includes generating one or more human-interpretable scores of an avatar.

### Claim 7 (depends on 1)

The method of claim 1, the step of combining the audio evaluation score and the video evaluation score comprises combining using at least one of a weighted average with fixed weights method and a trainable combination method.

### Claim 8 (depends on 7)

The method of claim 7, wherein the weighted average with fixed weights method comprises scaling all evaluation scores to a predefined range of weights and determining an average of the weights.

### Claim 9 (depends on 7)

The method of claim 7, wherein the trainable combination method comprises: using a dataset containing pairs of a video of the target person and corresponding mean opinion scores to train a regression module to predict the final score.

### Claim 10

An evaluation module for automated evaluation of an avatar generated by an avatar generator comprising:
a. an audio evaluator configured to:
i. obtain a speech generated by a TTS module of the avatar generator;
ii. obtain the audio features of the target person;
iii. compare the speech and the audio features using a set of audio metrics; and
iv. generate an audio evaluation score based on the comparison; and
b. a video evaluator configured to:
i. obtain a video clip generated by a video generator of the avatar generator;
ii. obtain the video features of the target person;
iii. compare the video clip with the video features using a set of video metrics; and
iv. generate a video evaluation score based on the comparison; and
c. a score combination module configured to:
i. combine the audio evaluation score and the video evaluation score;
ii. generate a combined naturalness score for the avatar generator based on the combination; and
iii. generate an overall naturalness score based on the naturalness score and the combined naturalness score.

### Claim 11 (depends on 10)

The system of claim 10, wherein the set of audio metrics comprises
a. ASR based evaluation metrics;
b. VAD based evaluation metrics;
c. GPE, FFE metrics; and
d. EER and COS metrics.

### Claim 12 (depends on 10)

The system of claim 10, wherein evaluation scores generated by each of the set of audio metrics are combined to generate the audio evaluation score.

### Claim 13 (depends on 10)

The system of claim 10, wherein the set of video metric includes:
a. PSNR, MS-SSIM, FSIM, LPIPS, VMAF, VIF, NLP metrics;
b. WaDIQaM, DUBCNN, TRES, ChipQA;
c. distribution-based metrics;
d. lip synchronization metrics; or
e. identity metrics.

### Claim 14 (depends on 10)

The system of claim 10, wherein the evaluation scores generated by each of the set of video metrics are combined to generate a video evaluation score.

### Claim 15 (depends on 10)

The system of claim 10, wherein the score combination module generates one or more human-interpretable scores of an avatar.

### Claim 16 (depends on 10)

The system of claim 10, wherein the score combination module is configured to combine the audio evaluation score and the video evaluation score using at least one of a weighted average with fixed weights method and a trainable combination method.

### Claim 17 (depends on 16)

The system of claim 16, wherein the score combination module using the weighted average with fixed weights method is configured to scale all evaluation scores to a predefined range of weights and determine an average of the weights.

### Claim 18 (depends on 16)

The system of claim 16, wherein the score combination module using the trainable combination method is configured to use a dataset containing pairs of a video of the target person and corresponding mean opinion scores to train a regression module to predict the final score.
