US 12,217,172 B2 · Adaptive off-ramp training and inference for early exits in a deep neural network

Abstract

Systems and methods are provided for training and using a deep neural network with adaptively trained off-ramps for an early exit at an intermediate representation layer. The training includes, for respective intermediate representation layers of a sequence of intermediate representation layers, predicting a label based on the training data and comparing against a correct label. The training further includes generating a confidence value associated with the predicted label. The confidence value is based on optimizing an objective function that includes a weighted entropy of a probability distribution of the likelihood, weighted based on whether previous intermediate representation layer has accurately predicted the label. Use of the weighted entropy provides the training with a focus on predicting labels that the previous intermediate representation layers has performed poorly and not labels that have existed before the intermediate representation layer being trained. Among alternative methods include a distilled twin, parallel neural network for predicting labels using adaptively trained off-ramps.

Claims20

Dependent claims

A computer-implemented method of training a deep neural network with an adaptive off-ramp for exiting the deep neural network, the method comprising:

receiving training data;
predicting, based on the training data, a label using a prediction layer in a sequence of prediction layers;
determining a combination of a weighted entropy value associated with the label and a confidence value of the label;
training, based on the combination and a predetermined threshold, the prediction layer and an off-ramp associated with the prediction layer to exit from the deep neural network;
removing, based at least on the trained prediction layer, a portion of the training data to create updated training data;
resampling the updated training data;
and
training, based on resampled training data, a subsequent production layer of the sequence of prediction layers.

5 more elements

A system for predicting a label using an adaptive off-ramp for an early exit from a deep neural network, the system comprises:

a processor;
and
a memory storing computer-executable instructions that when executed by the processor cause the system to:
receive an input query;
determine a feature of the input query;
determine, prior to processing the input query in the sequence of prediction layers and based on the feature of the input query, a target prediction layer to exit from the deep neural network, wherein the sequence of prediction layers includes a trained prediction layer;
iteratively predict, based on the input query, a candidate label using the sequence of trained prediction layers up to and including the target prediction layer of the deep neural network;
exit from the deep neural network using an off-ramp associated with the target prediction layer;
and
transmit the candidate label.

7 more elements

A computer storage media storing computer-executable instructions that when executed by a processor cause a computer system to perform operations comprising:

receiving an input query;
determining a feature of the input query;
determining, prior to processing the input query in a sequence of prediction layers and based on the feature of the input query, a target prediction layer to exit from a deep neural network, wherein the sequence of prediction layers includes a trained prediction layer, and wherein the trained prediction layer includes an intermediate representation layer of the deep neural network;
iteratively predicting, based on the input query, a candidate label using the sequence of trained prediction layers up to and including the target prediction layer of the deep neural network;
exiting the deep neural network using an off-ramp associated with the target prediction layer;
and
transmitting the candidate label.

4 more elements

Dependencies are inferred from the claim text.

Adaptive off-ramp training and inference for early exits in a deep neural network

Life of this patent

Adaptive off-ramp training and inference for early exits in a deep neural network

Life of this patent

Abstract

Claims20