# US 12,112,417 B2 · Artificial intelligence-based animation character drive method and related apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,417 B2
- Document ID: US12112417B2
- Status: Granted
- Kind: B2
- Application number: 18/080,655
- Filed: 2022-12-13
- Published: 2023-04-20
- Granted: 2024-10-08
- Earliest claimed: 2019-09-02
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Linchao Bao; Shiyin Kang; Sheng Wang; Xiangkai Lin; Xing Ji; Zhantu Zhu; Kuongchi Lei; Deyi Tuo; Peng Liu
- Web page: https://patentgrep.com/patent/US12112417B2

## CPC

- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- A63F 13/54: involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
- A63F 13/55: Controlling game characters or game objects based on the game progress
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06T 13/205: {driven by audio data}
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 40/174: {Facial expression recognition}
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 21/10: Transforming into visible information
- G10L 13/00: Speech synthesis; Text to speech systems
- G10L 2015/025: {Phonemes, fenemes or fenones being the recognition units}
- G10L 2021/105: {Synthesis of the lips movements from speech, e.g. for talking heads}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

This application discloses an artificial intelligence (AI) based animation character drive method. A first expression base of a first animation character corresponding to a speaker is determined by acquiring media data including a facial expression change when the speaker says a speech, and the first expression base may reflect different expressions of the first animation character. After target text information is obtained, an acoustic feature and a target expression parameter corresponding to the target text information are determined according to the target text information, the foregoing acquired media data, and the first expression base. A second animation character having a second expression base may be driven according to the acoustic feature and the target expression parameter, so that the second animation character may simulate the speaker's sound and facial expression when saying the target text information, thereby improving experience of interaction between the user and the animation character.

## Claims

### Claim 1

An animation character drive method performed by a computing device, the method comprising:
obtaining simultaneously captured multi-modal media data including a facial expression and a corresponding speech of a speaker;
determining a first expression base of a first animation character corresponding to the speaker according to the facial expression;
determining, according to the media data and the first expression base, an acoustic feature and a target expression parameter of the first animation character speaking target text information, the acoustic feature characterizing a sound of the speaker when speaking the target text information, and the target expression parameter characterizing a facial expression of the speaker when speaking the target text information based on the first expression base;
determining a mapping relationship between the first expression base of the first animation character and the second expression base of the second animation character, wherein an expression parameter of the first expression base is a function of an expression parameter of the second expression base; and
driving a second animation character having a second expression base to simulate a sound of the speaker speaking the target text information based on the acoustic feature and a facial expression of the second animation character based on the mapping relationship, the target expression parameter and the second expression base simultaneously.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the determining a first expression base of a first animation character corresponding to the speaker according to the facial expression comprises:
determining the first expression base of the first animation character and a face-to-parameter translation parameter of the first animation character according to the facial expression, the face-to-parameter translation parameter being used for identifying a change degree of a face shape of the first animation character relative to a face-to-parameter translation base corresponding to the first animation character.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the second expression base is generated according to a preset relationship between the second expression base and a phoneme, and the determining a mapping relationship between an expression parameter corresponding to the first expression base and an expression parameter corresponding to the second expression base comprises:
determining, according to the media data, a phoneme identified by the speech, a time interval corresponding to the phoneme, and video frames in which the media data is in the time interval;
determining a first expression parameter corresponding to the phoneme according to the video frames, the first expression parameter being used for identifying a change degree of a facial expression of the speaker when giving the phoneme relative to the first expression base;
determining a second expression parameter corresponding to the phoneme according to the preset relationship and the second expression base; and
determining the mapping relationship in a function form according to the first expression parameter and the second expression parameter.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the determining, according to the media data and the first expression base, an acoustic feature and a target expression parameter of the first animation character speaking target text information comprises:
determining, according to the target text information and the media data, the acoustic feature and an expression feature corresponding to the target text information, the expression feature identifying a facial expression of the speaker when the speaker says the target text information; and
determining the target expression parameter according to the first expression base and the expression feature.

### Claim 5

A computing device comprising a processor and a memory coupled to the processor, wherein the memory stores a plurality of computer programs that, when executed by the processor, cause the computing device to perform a plurality of operations including:
obtaining simultaneously captured multi-modal media data including a facial expression and a corresponding speech of a speaker;
determining a first expression base of a first animation character corresponding to the speaker according to the facial expression;
determining, according to the media data and the first expression base, an acoustic feature and a target expression parameter of the first animation character speaking target text information, the acoustic feature characterizing a sound of the speaker when speaking the target text information, and the target expression parameter characterizing a facial expression of the speaker when speaking the target text information based on the first expression base;
determining a mapping relationship between the first expression base of the first animation character and the second expression base of the second animation character, wherein an expression parameter of the first expression base is a function of an expression parameter of the second expression base; and
driving a second animation character having a second expression base to simulate a sound of the speaker speaking the target text information based on the acoustic feature and a facial expression of the second animation character based on the mapping relationship, the target expression parameter and the second expression base simultaneously.

### Claim 6 (depends on 5)

The computing device according to claim 5, wherein the determining a first expression base of a first animation character corresponding to the speaker according to the facial expression comprises:
determining the first expression base of the first animation character and a face-to-parameter translation parameter of the first animation character according to the facial expression, the face-to-parameter translation parameter being used for identifying a change degree of a face shape of the first animation character relative to a face-to-parameter translation base corresponding to the first animation character.

### Claim 7 (depends on 6)

The computing device according to claim 6, wherein the second expression base is generated according to a preset relationship between the second expression base and a phoneme, and the determining a mapping relationship between an expression parameter corresponding to the first expression base and an expression parameter corresponding to the second expression base comprises:
determining, according to the media data, a phoneme identified by the speech, a time interval corresponding to the phoneme, and video frames in which the media data is in the time interval;
determining a first expression parameter corresponding to the phoneme according to the video frames, the first expression parameter being used for identifying a change degree of a facial expression of the speaker when giving the phoneme relative to the first expression base;
determining a second expression parameter corresponding to the phoneme according to the preset relationship and the second expression base; and
determining the mapping relationship in a function form according to the first expression parameter and the second expression parameter.

### Claim 8 (depends on 5)

The computing device according to claim 5, wherein the determining, according to the media data and the first expression base, an acoustic feature and a target expression parameter of the first animation character speaking target text information comprises:
determining, according to the target text information and the media data, the acoustic feature and an expression feature corresponding to the target text information, the expression feature identifying a facial expression of the speaker when the speaker says the target text information; and
determining the target expression parameter according to the first expression base and the expression feature.

### Claim 9

A non-transitory computer-readable storage medium storing a plurality of computer programs that, when executed by a processor of a computing device, cause the computing device to perform a plurality of operations including:
obtaining simultaneously captured multi-modal media data including a facial expression and a corresponding speech of a speaker;
determining a first expression base of a first animation character corresponding to the speaker according to the facial expression;
determining, according to the media data and the first expression base, an acoustic feature and a target expression parameter of the first animation character speaking target text information, the acoustic feature characterizing a sound of the speaker when speaking the target text information, and the target expression parameter characterizing a facial expression of the speaker when speaking the target text information based on the first expression base;
determining a mapping relationship between the first expression base of the first animation character and the second expression base of the second animation character, wherein an expression parameter of the first expression base is a function of an expression parameter of the second expression base; and
driving a second animation character having a second expression base to simulate a sound of the speaker speaking the target text information based on the acoustic feature and a facial expression of the second animation character based on the mapping relationship, the target expression parameter and the second expression base simultaneously.

### Claim 10 (depends on 9)

The non-transitory computer-readable storage medium according to claim 9, wherein the determining a first expression base of a first animation character corresponding to the speaker according to the facial expression comprises:
determining the first expression base of the first animation character and a face-to-parameter translation parameter of the first animation character according to the facial expression, the face-to-parameter translation parameter being used for identifying a change degree of a face shape of the first animation character relative to a face-to-parameter translation base corresponding to the first animation character.

### Claim 11 (depends on 9)

The non-transitory computer-readable storage medium according to claim 9, wherein the second expression base is generated according to a preset relationship between the second expression base and a phoneme, and the determining a mapping relationship between an expression parameter corresponding to the first expression base and an expression parameter corresponding to the second expression base comprises:
determining, according to the media data, a phoneme identified by the speech, a time interval corresponding to the phoneme, and video frames in which the media data is in the time interval;
determining a first expression parameter corresponding to the phoneme according to the video frames, the first expression parameter being used for identifying a change degree of a facial expression of the speaker when giving the phoneme relative to the first expression base;
determining a second expression parameter corresponding to the phoneme according to the preset relationship and the second expression base; and
determining the mapping relationship according to the first expression parameter and the second expression parameter.

### Claim 12 (depends on 9)

The non-transitory computer-readable storage medium according to claim 9, wherein the determining, according to target text information, the media data, and the first expression base, an acoustic feature and a target expression parameter corresponding to the target text information comprises:
determining, according to the target text information and the media data, the acoustic feature and an expression feature corresponding to the target text information, the expression feature being used for identifying a facial expression that is simulated when the speaker says the target text information; and
determining the target expression parameter according to the first expression base and the expression feature.