# US 2019/0171868 A1 · Systems and methods for facial representation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0171868 A1
- Document ID: US20190171868A1
- Status: Published application
- Kind: A1
- Application number: 16/127,727
- Filed: 2018-09-11
- Published: 2019-06-06
- Earliest claimed: 2013-11-04
- Assignees: Facebook, Inc.
- Applicants: Facebook, Inc.
- Inventors: Yaniv Taigman; Ming Yang; Marc'Aurelio Ranzato
- Web page: https://patentgrep.com/patent/US20190171868A1

## CPC

- G06V 40/173: {face re-identification, e.g. recognising unknown faces across different face tracks}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/09: Supervised learning
- G06T 3/02: Affine transformations (for image registration G06T3/147; for image mosaicing G06T3/4038)
- G06T 3/18: Image warping, e.g. rearranging pixels individually
- G06T 15/205: {Image-based rendering}
- G06V 40/168: {Feature extraction; Face representation}
- G06V 40/172: {Classification, e.g. identification}
- G06T 2210/44: Morphing
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems, methods, and non-transitory computer readable media can align face images, classify face images, and verify face images by employing a deep neural network (DNN). A 3D-aligned face image can be generated from a 2D face image. An identity of the 2D face image can be classified based on provision of the 3D-aligned face image to the DNN. The identity of the 2D face image can comprise a feature vector.

## Claims

### Claim at position 1

-20. (canceled)

### Claim 21

A computer-implemented method, comprising:
determining, by a computing system, a set of fiducial points in a 2D face image;
generating, by the computing system, a 3D-aligned face image from the 2D face image based at least in part on the set of fiducial points determined for the 2D face image; and
classifying, by the computing system, a facial identity corresponding to the 2D face image based at least in part on the 3D-aligned face image.

### Claim 22 (depends on 21)

The computer-implemented method of claim 21, wherein generating the 3D-aligned face image from the 2D face image further comprises:
generating, by the computing system, a set of anchor points that are placed on a 3D shape of a human face, each anchor point corresponding to a fiducial point in the set of fiducial points determined for the 2D face image; and
transforming, by the computing system, the 2D face image to the 3D-aligned face image based on the set of fiducial points and the set of anchor points.

### Claim 23 (depends on 22)

The computer-implemented method of claim 22, wherein transforming the 2D face image to the 3D-aligned face image comprises a piece-wise affine warping of the 2D face image to the 3D-aligned face image based on the set of fiducial points and the set of anchor points.

### Claim 24 (depends on 22)

The computer-implemented method of claim 22, wherein the 3D shape of the human face is generated based on an average of a plurality of 3D scans of human faces.

### Claim 25 (depends on 21)

The computer-implemented method of claim 21, wherein classifying the facial identify corresponding to the 2D face image further comprises:
providing, by the computing system, the 3D-aligned face image as input to a deep neural network (DNN) that is trained to classify the facial identify corresponding to the 2D face image based on a feature vector representation of the 3D-aligned face image.

### Claim 26 (depends on 25)

The computer-implemented method of claim 25, wherein the deep neural network includes at least a convolutional pooling layer, a set of locally connected layers, and a set of fully connected layers, wherein the 3D-aligned face image is provided as input to the convolutional pooling layer, and wherein the set of fully connected layers output the feature vector representation of the 3D-aligned face image.

### Claim 27 (depends on 26)

The computer-implemented method of claim 26, wherein the convolutional pooling layer includes at least a first convolutional layer, a max pooling layer, and a second convolutional layer, wherein each convolutional layer includes a set of filters that each generate a feature map.

### Claim 28 (depends on 27)

The computer-implemented method of claim 27, wherein feature maps generated by the first convolutional layer are provided as inputs to the max pooling layer, and wherein outputs from the max pooling layer are provided as inputs to the second convolutional layer.

### Claim 29 (depends on 21)

The computer-implemented method of claim 21, wherein determining the set of fiducial points in the 2D face image further comprises:
determining, by the computing system, a face portion in the 2D face image; and
localizing, by the computing system, the set of fiducial points in the 2D face image.

### Claim 30 (depends on 21)

The computer-implemented method of claim 21, wherein the face portion in the 2D face image is determined based at least in part on a first machine learning regression model, and wherein the set of fiducial points in the 2D face image are localized based at least in part on a second machine learning regression model that is different from the first machine learning regression model.

### Claim 31

A system comprising:
at least one processor; and
a memory storing instructions that, when executed by the at least one processor, cause the system to perform:
determining a set of fiducial points in a 2D face image;
generating a 3D-aligned face image from the 2D face image based at least in part on the set of fiducial points determined for the 2D face image; and
classifying a facial identity corresponding to the 2D face image based at least in part on the 3D-aligned face image.

### Claim 32 (depends on 31)

The system of claim 31, wherein generating the 3D-aligned face image from the 2D face image further causes the system to perform:
generating a set of anchor points that are placed on a 3D shape of a human face, each anchor point corresponding to a fiducial point in the set of fiducial points determined for the 2D face image; and
transforming the 2D face image to the 3D-aligned face image based on the set of fiducial points and the set of anchor points.

### Claim 33 (depends on 32)

The system of claim 32, wherein transforming the 2D face image to the 3D-aligned face image comprises a piece-wise affine warping of the 2D face image to the 3D-aligned face image based on the set of fiducial points and the set of anchor points.

### Claim 34 (depends on 32)

The system of claim 32, wherein the 3D shape of the human face is generated based on an average of a plurality of 3D scans of human faces.

### Claim 35 (depends on 31)

The system of claim 31, wherein classifying the facial identify corresponding to the 2D face image further causes the system to perform:
providing the 3D-aligned face image as input to a deep neural network (DNN) that is trained to classify the facial identify corresponding to the 2D face image based on a feature vector representation of the 3D-aligned face image.

### Claim 36

A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform:
determining a set of fiducial points in a 2D face image;
generating a 3D-aligned face image from the 2D face image based at least in part on the set of fiducial points determined for the 2D face image; and
classifying a facial identity corresponding to the 2D face image based at least in part on the 3D-aligned face image.

### Claim 37 (depends on 36)

The non-transitory computer-readable storage medium of claim 36, wherein generating the 3D-aligned face image from the 2D face image further causes the system to perform:
generating a set of anchor points that are placed on a 3D shape of a human face, each anchor point corresponding to a fiducial point in the set of fiducial points determined for the 2D face image; and
transforming the 2D face image to the 3D-aligned face image based on the set of fiducial points and the set of anchor points.

### Claim 38 (depends on 37)

The non-transitory computer-readable storage medium of claim 37, wherein transforming the 2D face image to the 3D-aligned face image comprises a piece-wise affine warping of the 2D face image to the 3D-aligned face image based on the set of fiducial points and the set of anchor points.

### Claim 39 (depends on 37)

The non-transitory computer-readable storage medium of claim 37, wherein the 3D shape of the human face is generated based on an average of a plurality of 3D scans of human faces.

### Claim 40 (depends on 36)

The non-transitory computer-readable storage medium of claim 36, wherein classifying the facial identify corresponding to the 2D face image further causes the system to perform:
providing the 3D-aligned face image as input to a deep neural network (DNN) that is trained to classify the facial identify corresponding to the 2D face image based on a feature vector representation of the 3D-aligned face image.
