# US 11,908,240 B2 · Micro-expression recognition method based on multi-scale spatiotemporal feature neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,240 B2
- Document ID: US11908240B2
- Status: Granted
- Kind: B2
- Application number: 17/471,384
- Filed: 2021-09-10
- Published: 2022-08-25
- Granted: 2024-02-20
- Earliest claimed: 2021-02-19
- Assignees: Institute of Automation, Chinese Academy of Sciences
- Applicants: Institute of Automation, Chinese Academy of Sciences
- Inventors: Jianhua Tao; Hao Zhang; Bin Liu; Wenxiang She
- Web page: https://patentgrep.com/patent/US11908240B2

## CPC

- G06V 40/176: {Dynamic expression}
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V 40/168: {Feature extraction; Face representation}
- G06V 40/172: {Classification, e.g. identification}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Disclosed is a micro-expression recognition method based on a multi-scale spatiotemporal feature neural network, in which spatial features and temporal features of micro-expression are obtained from micro-expression video frames, and combined together to form more robust micro-expression features, at the same time, since the micro-expression occurs in local areas of a face, active local areas of the face during occurrence of the micro-expression and an overall area of the face are combined together for micro-expression recognition.

## Claims

### Claim 1

A micro-expression recognition method based on a multi-scale spatiotemporal feature neural network, comprising steps of:
S1, obtaining a collection of micro-expression videos, and converting the collection of micro-expression videos into a collection of micro-expression image frame sequence samples;
S2, extracting a set of face images from the collection of micro-expression image frame sequence samples, and performing gray-scale processing on the set of face images so as to form a set of face image sequence samples, wherein a face image in the set of face image sequence samples include an overall area of a human face;
S3, extracting local areas of interest of the face from the set of face image sequence samples to serve as samples including local features so as to form a set of local feature samples;
S4, normalizing time scales of each of the set of face image sequence samples and the set of local feature samples to obtain a normalized set of face image sequence samples and a normalized set of local feature samples, respectively;
S5, constructing a spatiotemporal feature neural network that is formed by merging a neural network for extracting spatial features and a neural network for extracting temporal features together, and taking the normalized set of face image sequence samples and the normalized set of local feature samples as input to train the spatiotemporal feature neural network so as to obtain a trained spatiotemporal feature neural network; and
S6, inputting a micro-expression video to be recognized into the trained spatiotemporal feature neural network to obtain recognized micro-expressions,
wherein the spatiotemporal feature neural network comprises a global feature extraction layer, a local feature extraction layer, a connection layer, a fully connected layer, and an output layer, wherein the global feature extraction layer and the local feature extraction layer are connected to the connection layer, respectively, the connection layer is connected to the fully connected layer, and the fully connected layer is connected to the output layer; and wherein each of the global feature extraction layer and the local feature extraction layer includes an input layer, a three-dimensional convolution layer, a three-dimensional maximum value pooling layer, a remodeling layer, and a long and short-term memory model recurrent neural network, which are connected to on another in sequence; and wherein global features of the spatiotemporal feature neural network relate to the overall area of the face, and local features of the spatiotemporal feature neural network relate to local areas of interest of the face.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the step S2 includes steps of:
S21, performing face detection on the collection of micro-expression image frame sequence samples, and cropping out rectangular face images to be zoomed in or out so as to obtain a face image sequence; and
S22: performing gray-scale normalization processing on the face image sequence according to a formula,
N
⁡
(
i
,
j
)
=
I
⁡
(
i
,
j
)
-
min
max
-
min
wherein I(i,j) and N(i,j) respectively represent a gray value of an original image of the face image sequence and a gray value of the image undergoing the gray normalization of the face image sequence, and max and min respectively represent maximum and minimum gray values of the original image of the face image sequence.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the step S3 includes steps of:
S31, selecting, according to face motion units, emotion labels in the set of face image sequence samples, and annotations regarding a single emotion label being associated with what face motion units, local areas where a motion amplitude exceeds a preset amplitude threshold during occurrence of a micro-expression as local areas of interest of the face; and
S32, cutting out the local areas of interest of the face from face images in the set of face image sequence samples to serve as samples including the local features of micro-expressions.

### Claim 4 (depends on 3)

The method according to claim 3, wherein each of the emotion labels is a micro-expression annotation that has been pre-labeled on a face image sequence sample.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the time scales of each of the set of face image sequence samples and the set of local feature samples is normalized by using a time interpolation algorithm.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the three-dimensional convolution layer and the fully connected layer each uses an activation function,
ReLU
⁡
(
x
)
=
{
0
,
x
≤
0
x
,
x
>
0
where x is an input variable of the activation function.

### Claim 7 (depends on 1)

The method according to claim 1, wherein a loss function used for training the spatiotemporal feature neural network is a cross-entropy loss function,
L
=
1
N
⁢
∑
i
L
i
=
1
N
⁢
∑
i
-
∑
c
=
1
M
y
ic
⁢
log
⁡
(
P
ic
)
where M is a quantity of categories; yic refers to an indicator variable, where if a current category is same as a category of an observed sample i, then yic is 1, otherwise yic is 0; Pic is a predicted probability of the observed sample i belonging to the current category; and N is a quantity of samples; and
wherein a Stochastic Gradient Descent method is used to calculate the loss function during training of the spatiotemporal feature neural network.

### Claim 8 (depends on 7)

The method according to claim 7, wherein a batch normalization algorithm is used during training of the spatiotemporal feature neural network; in the batch normalization algorithm, approximate whitening is performed by using a formula,
x
^
(
k
)
=
x
(
k
)
-
E
[
x
(
k
)
]
V
⁢
ar
[
x
(
k
)
]
where x(k) is an input of a certain neuron in a neural network layer, x(k)=Wh+b, W being a weight of the neural network layer, h being an output of the upper layer of the neural network layer, b being an uncertainty constant; E[x(k)] is a mean value of all input data in a batch of neurons of the neural network layer under the Stochastic Gradient Descent method; and √{square root over (Var[x(k)])} is a standard deviation of all input data in a batch of said neuron.

### Claim 9 (depends on 1)

A non-transitory computer-readable storage medium storing one or more programs thereon, wherein the one or more programs can be executed by one or more processors to implement the micro-expression recognition method based on the multi-scale spatiotemporal feature neural network according to claim 1.

### Claim 10 (depends on 1)

A computer device comprising a processor and a memory, wherein the memory is configured to store a computer program; the processor is configured to, when executing the computer program stored in the memory, implement the micro-expression recognition method based on the multi-scale spatiotemporal feature neural network according to claim 1.