# US 12,002,289 B2 · Expression recognition method and apparatus, computer device, and readable storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,289 B2
- Document ID: US12002289B2
- Status: Granted
- Kind: B2
- Application number: 17/434,424
- Filed: 2021-02-24
- Published: 2022-10-27
- Granted: 2024-06-04
- Earliest claimed: 2020-04-01
- Assignees: Boe Technology Group Co., Ltd.
- Applicants: BOE Technology Group Co., Ltd.
- Inventors: Guannan Chen
- Web page: https://patentgrep.com/patent/US12002289B2

## CPC

- G06V 40/174: {Facial expression recognition}
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06T 7/73: using feature-based methods
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/778: Active pattern-learning, e.g. online learning of image or video features
- G06V 10/806: {of extracted features}
- G06V 10/82: using neural networks
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06V 40/172: {Classification, e.g. identification}
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30201: Face
- Y02D 10/00: Energy efficient computing, e.g. low power processors, power management or thermal management
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- Y02D: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE

## Abstract

An expression recognition method and apparatus, a computer device, and a readable storage medium are provided. The method includes: performing face key-point position detection on a face image to obtain face key-point position information; and obtaining expression class information of the face image using four cascaded convolutional modules and a trained neural network classifier according to the face image and the face key-point position information.

## Claims

### Claim 1

An expression recognition method, comprising the following steps:
performing face key-point position detection on a face image to obtain face key-point position information;
inputting the face image to four cascaded convolutional modules, and sequentially performing feature processing on the input face image to obtain a feature response map output by a fourth convolutional module of the four cascaded convolutional modules;
inputting the feature response map output by the fourth convolutional module to a global average pooling layer module to obtain a feature vector of a first dimension;
performing key-point feature extraction on feature response maps output by the first three convolutional modules of the four cascaded convolutional modules respectively using the face key-point position information to obtain key-point feature information of the response feature maps output by the first three convolutional modules respectively;
connecting the feature vector of the first dimension with the key-point feature information of the feature response maps output by the first three convolutional modules respectively to obtain a feature vector of a second dimension;
inputting the feature vector of the second dimension to a fully connected layer module, to process the feature vector of the second dimension to obtain a feature vector of a third dimension; and
inputting the feature vector of the third dimension to a trained neural network classifier to output expression class information of the face image through the neural network classifier.

### Claim 2 (depends on 1)

The expression recognition method according to claim 1, wherein
the performing the face key-point position detection on the face image to obtain the face key-point position information comprises:
performing the face key-point position detection on the face image based on a Dlib library to acquire key-points of eyes and mouth in the face image as the face key-point position information.

### Claim 3 (depends on 2)

The expression recognition method according to claim 2, wherein
the neural network classifier is obtained by training through a stochastic gradient descent method.

### Claim 4 (depends on 3)

The expression recognition method according to claim 3, wherein the stochastic gradient descent method comprises:
acquiring a neural network to be trained and a plurality of face image samples preset with expression classes, acquiring and preprocessing a certain number of face image samples every time, inputting the preprocessed face image samples to the neural network to perform gradient descent iterative training until a preset training condition is satisfied, so as to obtain the trained neural network classifier.

### Claim 5 (depends on 4)

The expression recognition method according to claim 4, wherein
the preset training condition comprises that an iteration count reaches a preset count, or a value of a loss function is less than a preset value, wherein a cross entropy is used as the loss function.

### Claim 6 (depends on 1)

The expression recognition method according to claim 1, wherein each convolutional module comprises an input layer, a convolutional layer, a normalization layer, an activation function layer, a pooling layer, and an output layer; and
an input end of the convolutional layer is connected to the input layer, an input end of the normalization layer is connected to an output end of the convolutional layer, an input end of the activation function layer is connected to an output end of the normalization layer, an input end of the pooling layer is connected to an output end of the activation function layer, and an input end of the output layer is connected to an output end of the pooling layer.

### Claim 7 (depends on 1)

The expression recognition method according to claim 1, wherein
the performing the key-point feature extraction on the feature response maps output by the first three convolutional modules of the four cascaded convolutional modules respectively using the face key-point position information to obtain key-point feature information of the response feature maps output by the first three convolutional modules respectively comprises:
extracting response values corresponding to the face key-point position information from the feature response map output by each convolutional module using the face key-point position information; and
performing weighted averaging on the response values corresponding to the face key-point position information in each feature response map to obtain the key-point feature information of the feature maps output by the first three convolutional modules respectively.

### Claim 8 (depends on 7)

The expression recognition method according to claim 7, wherein
the key-point feature information is obtained through the following formula:
K
i
,
j
=
1
N
⁢
∑
n
=
1
N
⁢
K
i
,
j
n
,
where Ki,j is the key-point feature information, Ki,jn is a response value of the face key-point position information in an n-th channel in the feature response map, and N is the number of channels of the feature response map.

### Claim 9 (depends on 7)

The expression recognition method according to claim 7,
before the step of extracting the response values corresponding to the face key-point position information from the feature response map output by each convolutional module using the face key-point position information, the method further comprises:
adjusting a size of the feature response map output by each convolutional module to be the same as a size of the face image.

### Claim 10 (depends on 1)

The expression recognition method according to claim 1,
before the step of performing the face key-point position detection on the face image to obtain the face key-point position information, the method further comprises:
acquiring an input image, performing face detection on the input image, and adjusting a size of the detected face image to a preset size.

### Claim 11 (depends on 1)

The expression recognition method according to claim 1, wherein
the neural network classifier is trained by a stochastic gradient descent method.

### Claim 12 (depends on 1)

The expression recognition method according to claim 1, wherein the inputting the feature vector of the second dimension to the fully connected layer module, to process the feature vector of the second dimension to obtain the feature vector of the third dimension, comprises:
connecting each element of an output vector of the fully connected layer module with each element of the feature vector of the second dimension by fusing all features of the feature vector of the second dimension to obtain the feature vector of the third dimension.

### Claim 13 (depends on 12)

The expression recognition method according to claim 12, wherein
the third dimension is 1×128.

### Claim 14 (depends on 1)

The expression recognition method according to claim 1, wherein
inputting the feature vector of the third dimension to the trained neural network classifier to output the expression class information of the face image through the neural network classifier, comprises:
inputting the feature vector of the third dimension to a softmax layer in the trained neural network to calculate a degree of confidence of each preset expression class.

### Claim 15 (depends on 14)

The expression recognition method according to claim 14, wherein
the degree of confidence may be obtained through the following formula:
P
⁡
(
y
=
j
|
x
)
=
e
x
T
⁢
w
j
∑
k
=
1
K
⁢
e
x
T
⁢
w
k
wherein j is a sequence number of an expression class, x is the feature vector of the third dimension, w is a network weight parameter, and P(y=j|x) is a degree of confidence that a corresponding expression class is a j-th expression class when the input vector of the softmax layer is x.

### Claim 16 (depends on 14)

The expression recognition method according to claim 14, further comprising:
determining an expression class corresponding to the face image according to the degree of confidence of each expression class.

### Claim 17 (depends on 16)

The expression recognition method according to claim 16, wherein determining the expression class corresponding to the face image according to the degree of confidence of each expression class, comprises:
determining an expression class corresponding to a maximum degree of confidence as the expression class corresponding to the face image.

### Claim 18 (depends on 1)

A computer device, comprising a processor and a memory storing a computer program runnable in the processor, wherein the processor executes the program to implement the method according to claim 1.

### Claim 19 (depends on 1)

A non-transitory computer-readable storage medium in which instructions are stored, wherein when the computer-readable storage medium is run on a computer, the computer is enabled to execute the method according to claim 1.

### Claim 20

An expression recognition apparatus, comprising: a processor and a storage device, wherein the storage device is configured to store processor-executable programs, and the programs comprise:
a face key-point position detection module, configured to perform face key-point position detection on a face image to obtain face key-point position information;
four cascaded convolutional modules, configured to input the face image and sequentially perform feature processing on the input face image to obtain a feature response map output by a fourth convolutional module of the four cascaded convolutional modules;
a global average pooling layer module, configured to obtain a feature vector of a first dimension according to the input feature response map output by the fourth convolutional module;
a key-point feature information module, configured to perform key-point feature extraction on feature response maps output by the first three convolutional modules of the four cascaded convolutional modules respectively using the face key-point position information to obtain key-point feature information of the response feature maps output by the first three convolutional modules respectively;
a feature vector connecting module, configured to connect the feature vector of the first dimension with the key-point feature information of the feature response maps output by the first three convolutional modules respectively to obtain a feature vector of a second dimension;
a fully connected layer module, configured to process the input feature vector of the second dimension to obtain a feature vector of a third dimension; and
a neural network classifier, configured to input the feature vector of the third dimension to a trained neural network classifier to output expression class information of the face image through the neural network classifier.