# US 2019/0244014 A1 · All-in-one convolutional neural network for face analysis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0244014 A1
- Document ID: US20190244014A1
- Status: Published application
- Kind: A1
- Application number: 16/340,859
- Filed: 2017-10-10
- Published: 2019-08-08
- Earliest claimed: 2016-10-10
- Assignees: University of Maryland, College Park
- Applicants: University of Maryland, College Park
- Inventors: Rajeev RANJAN; Swaminathan SANKARANARAYANAN; Carlos CASTILLO; Ramalingam CHELLAPPA
- Web page: https://patentgrep.com/patent/US20190244014A1

## CPC

- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/24: Classification techniques
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V 40/172: {Classification, e.g. identification}
- G06V 40/174: {Facial expression recognition}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Various facial recognition systems may benefit from appropriate use of computer systems. For example, certain face analysis systems may benefit from an all-in-one convolutional neural network that has been appropriately configured. A method can include obtaining an image of a face. The method can also include processing the image of the face using a first set of convolutional network layers configured to perform subject-independent tasks. The method can further include subsequently processing the image of the face using a second set of convolutional network layers configured to perform subject-dependent tasks. The second set of convolutional network layers can be integrated with the first set of convolutional network layers to form a single convolutional neural network. The method can additionally include outputting facial image detection results based on the processing and subsequent processing.

## Claims

### Claim 1

A method, comprising:
obtaining an image of a face;
processing the image of the face using a first set of convolutional network layers configured to perform subject-independent tasks;
subsequently processing the image of the face using a second set of convolutional network layers configured to perform subject-dependent tasks, wherein the second set of convolutional network layers is integrated with the first set of convolutional network layers to form a single convolutional neural network; and
outputting facial image detection results based on the processing and subsequent processing.

### Claim 2 (depends on 1)

The method of claim 1, wherein the results comprise a plurality of facial detection, key point extraction, pose angle, smile expression, age, and gender.

### Claim 3 (depends on 1)

The method of claim 1, wherein the results comprise an identity descriptor corresponding to each detected face in the facial image detection results.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
comparing the identity descriptor to a stored identity descriptor; and
performing a face recognition or identity verification based on the comparison.

### Claim 5 (depends on 1)

The method of claim 1, wherein the first set of convolution layers comprise a fusion of first, third, and fifth convolutional layers.

### Claim 6 (depends on 5)

The method of claim 5, wherein in the first set of convolution layers the fusion is further attached to two convolutional layers and pooling layers, to obtain a feature map of size 6×6.

### Claim 7 (depends on 6)

The method of claim 6, wherein the first set of convolution layers further comprises a dimensionality reduction layer to reduce a number of feature maps to 256.

### Claim 8 (depends on 6)

The method of claim 6, wherein the dimensionality reduction layer is followed by a fully connected layer of dimension 2048, configured to form a generic representation of the subject-independent tasks.

### Claim 9 (depends on 1)

The method of claim 1, wherein the second set of convolutional network layers comprise fully connected layers of dimension 512 each.

### Claim 10 (depends on 9)

The method of claim 9, wherein the fully connected layers are followed by output layers.

### Claim 11

An apparatus, comprising:
at least one processor; and
at least one memory including computer program code,
wherein the at least one memory and the computer program code are configured to, with the least one processor, cause the apparatus at least to
obtain an image of a face;
process the image of the face using a first set of convolutional network layers configured to perform subject-independent tasks;
subsequently process the image of the face using a second set of convolutional network layers configured to perform subject-dependent tasks, wherein the second set of convolutional network layers is integrated with the first set of convolutional network layers to form a single convolutional neural network; and
output facial image detection results based on the processing and subsequent processing.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein the results comprise a plurality of facial detection, key point extraction, pose angle, smile expression, age, and gender.

### Claim 13 (depends on 11)

The apparatus of claim 11, wherein the results comprise an identity descriptor corresponding to each detected face in the facial image detection results.

### Claim 14 (depends on 11)

The apparatus of claim 11, wherein the at least one memory and the computer program code are further configured to, with the least one processor, cause the apparatus at least to:
compare the identity descriptor to a stored identity descriptor; and
perform a face recognition or identity verification based on the comparison.

### Claim 15 (depends on 11)

The apparatus of claim 11, wherein the first set of convolution layers comprise a fusion of first, third, and fifth convolutional layers.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein in the first set of convolution layers the fusion is further attached to two convolutional layers and pooling layers, to obtain a feature map of size 6×6.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein the first set of convolution layers further comprises a dimensionality reduction layer to reduce a number of feature maps to 256.

### Claim 18 (depends on 16)

The apparatus of claim 16, wherein the dimensionality reduction layer is followed by a fully connected layer of dimension 2048, configured to form a generic representation of the subject-independent tasks.

### Claim 19 (depends on 11)

The apparatus of claim 11, wherein the second set of convolutional network layers comprise fully connected layers of dimension 512 each.

### Claim 20 (depends on 19)

The apparatus of claim 19, wherein the fully connected layers are followed by output layers.

### Claim 21

A non-transitory computer-readable medium encoded with instructions that, when executed in hardware, perform a process, the process comprising:
obtaining an image of a face;
processing the image of the face using a first set of convolutional network layers configured to perform subject-independent tasks;
subsequently processing the image of the face using a second set of convolutional network layers configured to perform subject-dependent tasks, wherein the second set of convolutional network layers is integrated with the first set of convolutional network layers to form a single convolutional neural network; and
outputting facial image detection results based on the processing and subsequent processing.

### Claim 22 (depends on 21)

The non-transitory computer-readable medium of claim 21, wherein the results comprise a plurality of facial detection, key point extraction, pose angle, smile expression, age, and gender.

### Claim 23 (depends on 21)

The non-transitory computer-readable medium of claim 21, wherein the results comprise an identity descriptor corresponding to each detected face in the facial image detection results.

### Claim 24 (depends on 21)

The non-transitory computer-readable medium of claim 21, the process further comprising:
comparing the identity descriptor to a stored identity descriptor; and
performing a face recognition or identity verification based on the comparison.

### Claim 25 (depends on 21)

The non-transitory computer-readable medium of claim 21, wherein the first set of convolution layers comprise a fusion of first, third, and fifth convolutional layers.

### Claim 26 (depends on 25)

The non-transitory computer-readable medium of claim 25, wherein in the first set of convolution layers the fusion is further attached to two convolutional layers and pooling layers, to obtain a feature map of size 6×6.

### Claim 27 (depends on 26)

The non-transitory computer-readable medium of claim 26, wherein the first set of convolution layers further comprises a dimensionality reduction layer to reduce a number of feature maps to 256.

### Claim 28 (depends on 26)

The non-transitory computer-readable medium of claim 26, wherein the dimensionality reduction layer is followed by a fully connected layer of dimension 2048, configured to form a generic representation of the subject-independent tasks.

### Claim 29 (depends on 21)

The non-transitory computer-readable medium of claim 21, wherein the second set of convolutional network layers comprise fully connected layers of dimension 512 each.

### Claim 30 (depends on 29)

The non-transitory computer-readable medium of claim 29, wherein the fully connected layers are followed by output layers.