# US 10,373,332 B2 · Systems and methods for dynamic facial analysis using a recurrent neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,373,332 B2
- Document ID: US10373332B2
- Status: Granted
- Kind: B2
- Application number: 15/836,549
- Filed: 2017-12-08
- Published: 2019-06-13
- Granted: 2019-08-06
- Earliest claimed: 2017-12-08
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Jinwei Gu; Xiaodong Yang; Shalini De Mello; Jan Kautz
- Web page: https://patentgrep.com/patent/US10373332B2

## CPC

- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06T 7/73: using feature-based methods
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 40/167: {using comparisons between temporally consecutive images}
- G06T 3/4046: using neural networks
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30201: Face
- G06T 2207/30204: Marker
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method, computer readable medium, and system are disclosed for dynamic facial analysis. The method includes the steps of receiving video data representing a sequence of image frames including at least one head and extracting, by a neural network, spatial features comprising pitch, yaw, and roll angles of the at least one head from the video data. The method also includes the step of processing, by a recurrent neural network, the spatial features for two or more image frames in the sequence of image frames to produce head pose estimates for the at least one head.

## Claims

### Claim 1

A computer-implemented method for facial analysis, comprising:
transforming a fully-connected layer of a first neural network into a recurrent layer to produce a recurrent neural network (RNN), wherein, during training, the fully-connected layer learned a first weight matrix, and the recurrent layer uses the first weight matrix to process inputs to the recurrent layer and uses a second weight matrix to process hidden state produced by the recurrent layer for a previous time step;
receiving video data representing a sequence of image frames including at least one head;
extracting spatial features comprising pitch, yaw, and roll angles of the at least one head from the video data by a second neural network; and
processing, by the RNN, the spatial features for two or more image frames in the sequence of image frames to produce head pose estimates for the at least one head.

### Claim 2 (depends on 1)

The method of claim 1, wherein the spatial features are extracted for each image frame in the sequence of image frames.

### Claim 3 (depends on 1)

The method of claim 1, wherein the second neural network is trained using a first training dataset and the RNN is trained using a second training dataset.

### Claim 4 (depends on 1)

The method of claim 1, wherein the second neural network is a convolutional neural network (CNN).

### Claim 5 (depends on 1)

The method of claim 1, wherein the second neural network and the RNN are simultaneously trained using one training dataset.

### Claim 6 (depends on 1)

The method of claim 1, wherein the sequence of image frames includes facial landmarks associated with the at least one head and,
the neural network extracts additional spatial features from the video data; and
the RNN processes the additional spatial features for the two or more image frames in the sequence of image frames to produce facial landmark tracking data.

### Claim 7 (depends on 6)

The method of claim 6, wherein the facial landmark tracking data comprises three-dimensional positions.

### Claim 8 (depends on 1)

The method of claim 1, wherein the RNN is a fully connected RNN.

### Claim 9

A facial analysis system, comprising:
a first neural network configured to:
receive video data representing a sequence of image frames including at least one head;
extract spatial features comprising pitch, yaw, and roll angles of the at least one head from the video data; and
a recurrent neural network (RNN) that is coupled to the neural network and configured to process the spatial features for two or more image frames in the sequence of image frames to produce head pose tracking data for the at least one head, wherein a fully-connected layer of a second neural network is transformed into a recurrent layer to produce the RNN, the recurrent layer using a first weight matrix to process inputs to the recurrent layer and using a second weight matrix to process hidden state produced by the recurrent layer for a previous time step, and the first matrix is learned by the fully-connected layer during training.

### Claim 10 (depends on 9)

The facial analysis system of claim 9, wherein the spatial features are extracted for each image frame in the sequence of image frames.

### Claim 11 (depends on 9)

The facial analysis system of claim 9, wherein the first neural network is trained using a first training dataset and the RNN is trained using a second training dataset.

### Claim 12 (depends on 9)

The facial analysis system of claim 9, wherein the first neural network is a convolutional neural network (CNN).

### Claim 13 (depends on 9)

The facial analysis system of claim 9, wherein the first neural network and the RNN are simultaneously trained using one training dataset.

### Claim 14 (depends on 9)

The facial analysis system of claim 9, wherein the sequence of image frames includes facial landmarks associated with the at least one head and,
the first neural network is further configured to extract additional spatial features from the video data; and
the RNN is further configured to process the additional spatial features for the two or more image frames in the sequence of image frames to produce facial landmark tracking data.

### Claim 15 (depends on 14)

The facial analysis system of claim 14, wherein the facial landmark tracking data comprises three-dimensional positions.

### Claim 16 (depends on 9)

The facial analysis system of claim 9, wherein the RNN is a fully connected RNN.

### Claim 17 (depends on 9)

The facial analysis system of claim 9, wherein the video data comprises color values.

### Claim 18

A non-transitory computer-readable media storing computer instructions for facial analysis that, when executed by one or more processors, cause the one or more processors to perform the steps of:
transforming a fully-connected layer of a first neural network into a recurrent layer to produce a recurrent neural network (RNN), wherein, during training, the fully-connected layer learned a first weight matrix, and the recurrent layer uses the first weight matrix to process inputs to the recurrent layer and uses a second weight matrix to process hidden state produced by the recurrent layer for a previous timestep;
receiving video data representing a sequence of image frames including at least one head;
extracting spatial features comprising pitch, yaw, and roll angles of the at least one head from the video data by a second neural network; and
processing, by the RNN, the spatial features for two or more image frames in the sequence of image frames to produce head pose estimates for the at least one head.

### Claim 19 (depends on 1)

The method of claim 1, wherein, during additional training, the second weight matrix is learned by the RNN.

### Claim 20 (depends on 5)

The method of claim 5, wherein the training dataset comprises a synthetic head pose dataset comprising video images and ground truth annotations.