Efficient distributed trainer with gradient accumulation on sampled weight for deep neural networks in facial recognition
AssigneeClearview AI, Inc.
InventorZhi Liu
Life of this patent
Estimating the term…
Abstract
This disclosure provides a highly scalable training data preparation pipeline for data cleaning and augmentation with the aim of extracting the most meaningful information while keeping the noise level low, as well as a highly efficient distributed trainer for the deep neural networks suitable for facial recognition. The goal is to train deeper and larger neural networks with larger and higher quality facial image datasets iteratively and frequently without incurring prohibitive costs and drastic delays.
Claims18
A method for training a neural network facial recognition model at scale, comprising:
- (a) receiving a batch of facial image data and dividing the batch into two or more parts;
- (b) performing label synchronization and classification layer sampling using full batch labels and keeping track of the normalized weight as a result of the sampling process;
- (c) forwarding a first part of the batch through a neural network backbone distributed in a plurality of GPUs;
- (d) synchronizing resulting feature vectors and splitting previously synchronized labels to obtain corresponding labels for the first part of the batch and running forward-backward with sampled weight to compute loss and gradients for sampled weight and feature vectors;
- (e) running backward from the feature vectors with the gradients to back-propagate and derive gradients for backbone parameters in each layer;
- (f) performing steps (c) to (e) for the part of the two or remaining parts of the batch to allow gradients to accumulate over the first pass for both the sampled weight and the backbone parameters;
- (g) synchronizing all the backbone parameters across all of the plurality of GPUs;
- (h) running an optimizer for the sample weight and the backbone parameters;
- and
- (i) assigning back the sampled weight and optimizer states to the classification layer.
A system for training a neural network facial recognition model at scale, comprising:
- a non-transitory, computer-readable memory;
- one or more processors;
- and
- a computer-readable medium containing programming instructions that, when executed by the one or more processors, configure the system to:
- (a) receive a batch of facial image data and dividing the batch into two or more parts;
- (b) perform label synchronization and classification layer sampling using full batch labels and keeping track of the normalized weight as a result of the sampling process;
- (c) forward a first part of the batch through a neural network backbone distributed in a plurality of GPUs;
- (d) synchronize resulting feature vectors and splitting previously synchronized labels to obtain corresponding labels for the first part of the batch and running forward-backward with sampled weight to compute loss and gradients for sampled weight and feature vectors;
- (e) run backward from the feature vectors with the gradients to back-propagate and derive gradients for backbone parameters in each layer;
- (f) perform steps (c) to (e) for the part of the two or remaining parts of the batch to allow gradients to accumulate over the first pass for both the sampled weight and the backbone parameters;
- (g) synchronize all the backbone parameters across all of the plurality of GPUs;
- (h) run an optimizer for the sample weight and the backbone parameters;
- and
- (i) assign back the sampled weight and optimizer states to the classification layer.
Dependencies are inferred from the claim text.