US 2025/0094813 A1Published application

Training a transformer neural network to perform task-specific parameter selection

AssigneeNVIDIA Corporation

InventorsWonmin BYEON, Sudarshan BABU, Shalini DE MELLO, Jan KAUTZ

Life of this patent

Estimating the term…

Loading patent…

Abstract

One embodiment of the present invention sets forth a technique for training a transformer neural network. The technique includes inputting a first task token and a first set of samples into the transformer neural network and training the transformer neural network using a first set of losses between predictions generated by the transformer neural network from the first task token and first set of samples as well as a first set of labels. The technique also includes converting the first task token into a second task token that is larger than the first task token, inputting the second task token and a second set of samples into the transformer neural network, and training the transformer neural network using a second set of losses between predictions generated by the transformer neural network from the second task token and the second set of samples as well as a second set of labels.

Claims20

Dependent claims

A computer-implemented method for training a transformer neural network, the method comprising:

inputting a first task token and a first set of samples included in a training dataset into the transformer neural network;
training the transformer neural network based on a first set of losses computed between a first set of predictions generated by the transformer neural network from the first task token and the first set of samples as well as a first set of labels for the first set of samples;
converting the first task token into a second task token that is larger than the first task token;
inputting the second task token and a second set of samples included in the training dataset into the transformer neural network;
and
performing a plurality of training operations on the transformer neural network based on a second set of losses computed between a second set of predictions generated by the transformer neural network from the second task token and the second set of samples as well as a second set of labels for the second set of samples, wherein, once trained, the transformer neural network generates a third task token from at least a portion of a data sample and performs a task associated with the data sample based on the third task token.

3 more elements

One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:

inputting a first task token and a first set of samples included in a training dataset into a transformer neural network;
training the transformer neural network based on a first set of losses computed between a first set of predictions generated by the transformer neural network from the first task token and the first set of samples as well as a first set of labels for the first set of samples;
converting the first task token into a second task token that is larger than the first task token;
inputting the second task token and a second set of samples included in the training dataset into the transformer neural network;
and
training the transformer neural network based on a second set of losses computed between a second set of predictions generated by the transformer neural network from the second task token and the second set of samples as well as a second set of labels for the second set of samples, wherein, once trained, the transformer neural network generates a third task token from at least a portion of a data sample and performs a task associated with the data sample based on the third task token.

3 more elements

Dependencies are inferred from the claim text.