# US 2022/0036636 A1 · Three-dimensional expression base generation method and apparatus, speech interaction method and apparatus, and medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036636 A1
- Document ID: US20220036636A1
- Status: Published application
- Kind: A1
- Application number: 17/503,178
- Filed: 2021-10-15
- Published: 2022-02-03
- Earliest claimed: 2019-11-15
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Xiangkai LIN; Linchao BAO
- Web page: https://patentgrep.com/patent/US20220036636A1

## CPC

- G06T 15/04: Texture mapping
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 7/33: using feature-based methods
- G06T 7/55: from multiple images
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 13/205: {driven by audio data}
- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G06V 20/647: {by matching two-dimensional images to three-dimensional objects}
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G10L 21/10: Transforming into visible information
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30196: Human being; Person
- G06T 2207/30201: Face
- G06T 2219/2004: Aligning objects, relative positioning of parts
- G10L 2021/105: {Synthesis of the lips movements from speech, e.g. for talking heads}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

This application provides a three-dimensional (3D) expression base generation method performed by a computer device. The method includes: obtaining image pairs of a target object in n types of head postures, each image pair including a color feature image and a depth image in a head posture; constructing a 3D human face model of the target object according to then image pairs; and generating a set of expression bases of the target object according to the 3D human face model of the target object. According to this application, based on a reconstructed 3D human face model, a set of expression bases of a target object is further generated, so that more diversified product functions may be expanded based on the set of expression bases.

## Claims

### Claim 1

A computer-implemented method performed by a computer device, the method comprising:
obtaining n sets of image pairs of a target object inn types of head postures, the n sets of image pairs comprising color feature images and depth images in the n types of head postures, an ith head posture being corresponding to an ith set of image pair, n being a positive integer, 0<i≤n;
constructing a 3D human face model of the target object according to the n sets of image pairs; and
generating an expression base corresponding to the target object based on the 3D human face model.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the constructing a 3D human face model of the target object according to the n sets of image pairs comprises:
obtaining head posture information and key point information respectively corresponding to the n sets of image pairs;
fusing the head posture information and the key point information to obtain a point cloud corresponding to a head of the target object; and
generating the 3D human face model of the target object according to the point cloud.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the obtaining head posture information and key point information respectively corresponding to then sets of image pairs comprises:
obtaining a key point in the color feature image of a kth set of image pair in the n sets of image pairs, 0<k≤n;
obtaining the key point information corresponding to the key point according to the depth image of the kth set of image pair, the key point information comprising 3D position information of the key point in the depth image; and
determining the head posture information corresponding to the kth set of image pair according to the key point information.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the generating an expression base corresponding to the target object based on the 3D human face model comprises:
performing preliminary point cloud alignment on the 3D human face model and a benchmark 3D human face model of a standard object;
attaching a point cloud of the standard object to the point cloud of the target object to obtain a low-model point cloud of the target object; and
generating the expression base corresponding to the target object according to the low-model point cloud.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the generating the expression base corresponding to the target object according to the low-model point cloud comprises:
determining a shape coefficient and an expression coefficient corresponding to the target object according to the low-model point cloud, the shape coefficient being used for controlling an appearance of the target object, and the expression coefficient being used for controlling an expression of the target object; and
adjusting the expression coefficient of the target object to obtain the expression base corresponding to the target object.

### Claim 6 (depends on 1)

The method according to claim 1, wherein after the generating an expression base corresponding to the target object based on the 3D human face model, the method further comprises:
extracting a texture map of the target object from the color feature images of the n sets of image pairs; and
generating a 3D human face model with texture of the target object in a target expression according to the expression base and the texture map, the target expression being corresponding to the expression base.

### Claim 7 (depends on 6)

The method according to claim 6, wherein the generating a 3D human face model with texture of the target object in a target expression according to the expression base and the texture map comprises:
determining a target expression coefficient corresponding to the target expression;
generating a 3D human face model of the target object in the target expression according to the target expression coefficient and the expression base; and
rendering the 3D human face model in the target expression by using the texture map to obtain the 3D human face model with texture in the target expression.

### Claim 8 (depends on 1)

The method according to claim 1, wherein the obtaining n sets of image pairs of a target object in n types of head postures comprises:
obtaining candidate image pairs of the target object;
sifting, from the candidate image pairs and according to a quality requirement, target image pairs meeting the quality requirement; and
selecting the image pairs corresponding to the n types of head postures from the target image pairs.

### Claim 9 (depends on 1)

The method according to claim 1, further comprising:
determining an expression sequence corresponding to speech information to be played, the expression sequence comprising at least one expression;
generating 3D human face models of the target object using the expression base corresponding to the target object and the expression sequence corresponding to the speech information; and
displaying the 3D human face models in turn according to a sequence of the at least one expression in the expression sequence in a process of playing the speech information.

### Claim 10 (depends on 9)

The method according to claim 9, wherein after the generating 3D human face models of the target object using the expression base corresponding to the target object and the expression sequence corresponding to the speech information, the method further comprises:
rendering the 3D human face models by using a texture map of the target object; and
the displaying the 3D human face models in turn according to a sequence of the at least one expression in the expression sequence in a process of playing the speech information comprises:
displaying the 3D human face models with texture in turn according to the sequence of the at least one expression in the expression sequence in the process of playing the speech information.

### Claim 11

A computer device, comprising a processor and a memory, the memory storing at least one program, the at least one program being loaded and executed by the processor to perform a plurality of operations including:
obtaining n sets of image pairs of a target object inn types of head postures, the n sets of image pairs comprising color feature images and depth images in the n types of head postures, an ith head posture being corresponding to an ith set of image pair, n being a positive integer, 0<i≤n;
constructing a 3D human face model of the target object according to the n sets of image pairs; and
generating an expression base corresponding to the target object based on the 3D human face model.

### Claim 12 (depends on 11)

The computer device according to claim 11, wherein the constructing a 3D human face model of the target object according to the n sets of image pairs comprises:
obtaining head posture information and key point information respectively corresponding to the n sets of image pairs;
fusing the head posture information and the key point information to obtain a point cloud corresponding to a head of the target object; and
generating the 3D human face model of the target object according to the point cloud.

### Claim 13 (depends on 12)

The computer device according to claim 12, wherein the obtaining head posture information and key point information respectively corresponding to the n sets of image pairs comprises:
obtaining a key point in the color feature image of a kth set of image pair in the n sets of image pairs, 0<k≤n;
obtaining the key point information corresponding to the key point according to the depth image of the kth set of image pair, the key point information comprising 3D position information of the key point in the depth image; and
determining the head posture information corresponding to the kth set of image pair according to the key point information.

### Claim 14 (depends on 11)

The computer device according to claim 11, wherein the generating an expression base corresponding to the target object based on the 3D human face model comprises:
performing preliminary point cloud alignment on the 3D human face model and a benchmark 3D human face model of a standard object;
attaching a point cloud of the standard object to the point cloud of the target object to obtain a low-model point cloud of the target object; and
generating the expression base corresponding to the target object according to the low-model point cloud.

### Claim 15 (depends on 14)

The computer device according to claim 14, wherein the generating the expression base corresponding to the target object according to the low-model point cloud comprises:
determining a shape coefficient and an expression coefficient corresponding to the target object according to the low-model point cloud, the shape coefficient being used for controlling an appearance of the target object, and the expression coefficient being used for controlling an expression of the target object; and
adjusting the expression coefficient of the target object to obtain the expression base corresponding to the target object.

### Claim 16 (depends on 11)

The computer device according to claim 11, wherein the plurality of operations further comprise:
extracting a texture map of the target object from the color feature images of the n sets of image pairs; and
generating a 3D human face model with texture of the target object in a target expression according to the expression base and the texture map, the target expression being corresponding to the expression base.

### Claim 17 (depends on 16)

The computer device according to claim 16, wherein the generating a 3D human face model with texture of the target object in a target expression according to the expression base and the texture map comprises:
determining a target expression coefficient corresponding to the target expression;
generating a 3D human face model of the target object in the target expression according to the target expression coefficient and the expression base; and
rendering the 3D human face model in the target expression by using the texture map to obtain the 3D human face model with texture in the target expression.

### Claim 18 (depends on 11)

The computer device according to claim 11, wherein the obtaining n sets of image pairs of a target object inn types of head postures comprises:
obtaining candidate image pairs of the target object;
sifting, from the candidate image pairs and according to a quality requirement, target image pairs meeting the quality requirement; and
selecting the image pairs corresponding to the n types of head postures from the target image pairs.

### Claim 19 (depends on 11)

The computer device according to claim 11, wherein the plurality of operations further comprise:
determining an expression sequence corresponding to speech information to be played, the expression sequence comprising at least one expression;
generating 3D human face models of the target object using the expression base corresponding to the target object and the expression sequence corresponding to the speech information; and
displaying the 3D human face models in turn according to a sequence of the at least one expression in the expression sequence in a process of playing the speech information.

### Claim 20

A non-transitory computer-readable storage medium, storing at least one program, the at least one program being loaded and executed by a processor of a computer device to perform a plurality of operations including:
obtaining n sets of image pairs of a target object inn types of head postures, the n sets of image pairs comprising color feature images and depth images in the n types of head postures, an ith head posture being corresponding to an ith set of image pair, n being a positive integer, 0<i≤n;
constructing a 3D human face model of the target object according to the n sets of image pairs; and
generating an expression base corresponding to the target object based on the 3D human face model.