Attention-based decoder-only sequence transduction neural networks

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training a self-attention decoder neural network for processing an input sequence comprising a plurality of input tokens to generate an output sequence comprising a plurality of output tokens, wherein the self-attention decoder neural network comprises a plurality of neural network layers that include a plurality of masked self-attention neural network layers, the operations comprising, for each of a plurality of iterations:

• obtaining one or more training examples, wherein each training example comprises a respective training input sequence and a respective training output sequence;
• for each respective training example, processing using the self-attention decoder neural network according to current values of parameters of the self-attention decoder neural network, a combined sequence comprising (i) tokens in the respective training input sequence and (ii) tokens in the respective training output sequence to generate a respective prediction output that comprises a time step output for each of the tokens in the respective training output sequence;
• and
• updating the current values of the parameters of the self-attention decoder neural network based on (i) the respective training output sequences and (ii) the respective prediction outputs.

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a self-attention decoder neural network for processing an input sequence comprising a plurality of input tokens to generate an output sequence comprising a plurality of output tokens, wherein the self-attention decoder neural network comprises a plurality of neural network layers that include a plurality of masked self-attention neural network layers, the operations comprising, for each of a plurality of iterations:

• obtaining one or more training examples, wherein each training example comprises a respective training input sequence and a respective training output sequence;
• for each respective training example, processing using the self-attention decoder neural network according to current values of parameters of the self-attention decoder neural network, a combined sequence comprising (i) tokens in the respective training input sequence and (ii) tokens in the respective training output sequence to generate a respective prediction output that comprises a time step output for each of the tokens in the respective training output sequence;
• and
• updating the current values of the parameters of the self-attention decoder neural network based on (i) the respective training output sequences and (ii) the respective prediction outputs.