# US 2023/0267709 A1 · Dataset-aware and invariant learning for face recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0267709 A1
- Document ID: US20230267709A1
- Status: Published application
- Kind: A1
- Application number: 18/004,850
- Filed: 2021-07-09
- Published: 2023-08-24
- Earliest claimed: 2020-07-09
- Assignees: Wyze Labs, Inc.
- Applicants: Wyze Labs, Inc.
- Inventors: Lin Chen; Gaoang Wang; Tianqiang Liu
- Web page: https://patentgrep.com/patent/US20230267709A1

## CPC

- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/048: Activation functions
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06V 10/776: Validation; Performance evaluation
- G06V 10/82: using neural networks
- G06V 30/19147: {Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting}
- G06V 40/168: {Feature extraction; Face representation}
- G06V 40/172: {Classification, e.g. identification}
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Introduced here is an approach to training a model to perform facial recognition with a designed dataset-aware loss. This approach is particularly useful for large-scale, multi-dataset training since there are no prerequisites for label cleaning. Dataset-aware loss may be built upon the softmax function with a binary dataset indicator. Such an approach allows the dataset-aware loss to be readily combined with other losses to improve performance.

## Claims

### Claim 1

A non-transitory medium with instructions stored thereon that, when executed by a processor of a computing device, cause the computing device to perform operations comprising:
obtaining
(i) a first dataset that includes a first series of facial images of a first set of individuals, and
(ii) a second dataset that includes a second series of facial images of a second set of individuals that shares at least one individual in common with the first set of individuals;
for each facial image in the first dataset,
computing an embedding and then determining loss for the embedding within the first dataset, so as to produce a first series of loss metrics;
for each facial image in the second dataset,
computing an embedding and then determining loss for the embedding within the second dataset, so as to produce a second series of loss metrics; and
storing the first and second series of loss metrics in a data structure that is associated with a model to be trained to perform facial recognition using the first and second datasets.

### Claim 2 (depends on 1)

The non-transitory medium of claim 1, wherein the model is representative of a neural network that includes (i) an embedding network, (ii) a classification network, and (iii) a dataset classifier network.

### Claim 3 (depends on 2)

The non-transitory medium of claim 2, wherein the dataset classifier network is configured to classify whether a given facial image provided to the embedding network as input belongs to the first dataset or the second dataset.

### Claim 4 (depends on 2)

The non-transitory medium of claim 2,
wherein the embedding network takes, as input, a given facial image and a first network parameter and then produces a given embedding as output, and
wherein the classification network takes, as input, the given embedding and a second network parameter and then produces a proposed classification for the given facial image as output, and
wherein the dataset classifier network takes, as input, the given embedding and a third network parameters and then produces a mapping of the proposed classification to a given dataset to which the given facial image belongs.

### Claim 5 (depends on 4)

The non-transitory medium of claim 4, wherein the operations further comprise:
learning the first, second, and/or third network parameters by identifying embeddings that minimize loss as much as possible.

### Claim 6 (depends on 5)

The non-transitory medium of claim 5, wherein said learning further comprises optimizing the first, second, and/or third network parameters in a multi-stage process to ensure that embeddings are dataset invariant.

### Claim 7 (depends on 6)

The non-transitory medium of claim 6,
wherein in a first stage of the multi-stage process, the second and third network parameters are initialized and then the classification network and the dataset classifier network are trained separately, and
wherein in a second stage of the multi-stage process, the second and third network parameters are tuned following convergence of the classification network and the dataset classifier network.

### Claim 8 (depends on 1)

The non-transitory medium of claim 1,
wherein each loss metric in the first series of loss metrics is determined for a corresponding facial image in the first series of facial images, and
wherein each loss metric in the second series of loss metrics is determined for a corresponding facial image in the second series of facial images.

### Claim 9

A method comprising:
forming, by a processor, a training set from multiple datasets of facial images;
generating, by the processor, a feature vector for each facial image in the training set using a model,
wherein each feature vector includes
(i) an embedding of features of a face that are determined through analysis of a corresponding facial image,
(ii) an identity label that specifies an identity of an individual to whom the face belongs, and
(iii) an index label that specifies a dataset of which the corresponding facial image is a part;
determining, by the processor, loss for each feature vector entirely within a corresponding dataset, such that loss is separately determined within each dataset of the multiple datasets; and
propagating, by the processor, the losses to the model, so as to produce a trained model that is able to perform facial recognition; and
storing, by the processor, the trained model in a data structure.

### Claim 10 (depends on 8)

The method of claim 8,
wherein the model includes
(i) an embedding network that produces the embedding as output,
(ii) a classification network that produces the identity label as output and determines a first classification loss for the embedding, and
(iii) a dataset classifier network that produces the index label as output and determines a second classification loss for the embedding,
wherein said propagating comprises:
backpropagating the first classification loss through the classification network while modifying the classification network to train the classification network; and
backpropagating the second classification loss through the dataset classifier network while modifying the dataset classifier network to train the dataset classifier network.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
after backpropagating the first classification loss through the classification network,
backpropagating the first classification loss through the embedding network while modifying the embedding network to train the embedding network; and
after backpropagating the second classification loss through the dataset classifier network,
backpropagating the second classification loss through the embedding network while modifying the embedding network to train the embedding network.

### Claim 12 (depends on 9)

The method of claim 9, wherein the multiple datasets include (i) a first set of facial images that includes a first facial image of a given individual and (ii) a second set of facial images includes a second facial image of the given individual.

### Claim 13 (depends on 12)

The method of claim 12, wherein the first and second facial images are different facial images.

### Claim 14 (depends on 8)

The method of claim 8, wherein each dataset of the multiple datasets is obtained from a different source.

### Claim 15

A system for training a model to perform facial recognition, the system comprising:
an embedding network that takes, as input, an image of a face to be recognized and then outputs an embedding;
a classification network that takes, as input, the embedding and then outputs a first label that is representative of a proposed identity of an individual to whom the face belongs; and
a dataset classifier network that takes, as input, the embedding and then outputs a second label that is representative of a mapping of the first label to a dataset to which the image belongs.

### Claim 16 (depends on 15)

The system of claim 15, wherein the embedding is a vector representation of facial features that are extracted from the image by the embedding network.

### Claim 17 (depends on 15)

The system of claim 15,
wherein the embedding network further takes, as input, a first network parameter, and
wherein the classification network further takes, as input, a second network parameter.

### Claim 18 (depends on 17)

The system of claim 17, wherein the first and second network parameters are learned in training through analysis of embeddings, output by the embedding network for facial images that serve as training data, that minimize classification loss.

### Claim 19 (depends on 15)

The system of claim 15,
wherein the classification network is further configured to determine a dataset-aware classification loss for the embedding, and
wherein the dataset classifier network is further configured to determine a classification loss for the embedding.

### Claim 20 (depends on 19)

The system of claim 19, further comprising:
a backpropagation algorithm that, when executed,
backpropagates the dataset-aware classification loss to the classification network and the embedding network, and
backpropagates the classification loss to the dataset classifier network and the embedding network.
