Micro-expression recognition method based on multi-scale spatiotemporal feature neural network

A micro-expression recognition method based on a multi-scale spatiotemporal feature neural network, comprising steps of:

• S1, obtaining a collection of micro-expression videos, and converting the collection of micro-expression videos into a collection of micro-expression image frame sequence samples;
• S2, extracting a set of face images from the collection of micro-expression image frame sequence samples, and performing gray-scale processing on the set of face images so as to form a set of face image sequence samples, wherein a face image in the set of face image sequence samples include an overall area of a human face;
• S3, extracting local areas of interest of the face from the set of face image sequence samples to serve as samples including local features so as to form a set of local feature samples;
• S4, normalizing time scales of each of the set of face image sequence samples and the set of local feature samples to obtain a normalized set of face image sequence samples and a normalized set of local feature samples, respectively;
• S5, constructing a spatiotemporal feature neural network that is formed by merging a neural network for extracting spatial features and a neural network for extracting temporal features together, and taking the normalized set of face image sequence samples and the normalized set of local feature samples as input to train the spatiotemporal feature neural network so as to obtain a trained spatiotemporal feature neural network;
• and
• S6, inputting a micro-expression video to be recognized into the trained spatiotemporal feature neural network to obtain recognized micro-expressions,
• wherein the spatiotemporal feature neural network comprises a global feature extraction layer, a local feature extraction layer, a connection layer, a fully connected layer, and an output layer, wherein the global feature extraction layer and the local feature extraction layer are connected to the connection layer, respectively, the connection layer is connected to the fully connected layer, and the fully connected layer is connected to the output layer;
• and wherein each of the global feature extraction layer and the local feature extraction layer includes an input layer, a three-dimensional convolution layer, a three-dimensional maximum value pooling layer, a remodeling layer, and a long and short-term memory model recurrent neural network, which are connected to on another in sequence;
• and wherein global features of the spatiotemporal feature neural network relate to the overall area of the face, and local features of the spatiotemporal feature neural network relate to local areas of interest of the face.