Training machine-learned transformer architectures by clipping queries, keys, and values

A non-transitory computer readable storage medium comprising stored program code, the program code comprising instructions, the instructions when executed causes a processor system to:

• access a transformer architecture including a set of decoders coupled to receive a set of inputs and generate a set of outputs, wherein at least one decoder includes an attention block, the attention block coupled to receive a query, a key, and a value and generate an attention output;
• for one or more iterations:
• obtain a batch of training instances for a current iteration,
• apply parameters of the transformer architecture for the current iteration to a set of inputs obtained from the batch of training instances to generate a set of estimated outputs, the applying comprising for an attention block:
• obtaining a query, a key, and a value from the set of inputs, and
• applying a clipping operation to values of the query, the key, and the value,
• determine a loss function indicating a difference between data in the batch of training instances and the set of estimated outputs, and
• backpropagate terms obtained from the loss function to update the parameters of the transformer architecture;
• and
• deploy the trained transformer architecture to an inference system.

A computer system, comprising:

• a computer processor;
• and
• a non-transitory computer readable storage medium comprising stored instructions that when executed by the computer processor, cause the computer system to:
• access a transformer architecture including a set of decoders coupled to receive a set of inputs and generate a set of outputs, wherein at least one decoder includes an attention block, the attention block coupled to receive a query, a key, and a value and generate an attention output;
• for one or more iterations:
• obtain a batch of training instances for a current iteration,
• apply parameters of the transformer architecture for the current iteration to a set of inputs obtained from the batch of training instances to generate a set of estimated outputs, the applying comprising for an attention block:
• obtaining a query, a key, and a value from the set of inputs, and
• applying a clipping operation to values of the query, the key, and the value,
• determine a loss function indicating a difference between data in the batch of training instances and the set of estimated outputs, and
• backpropagate terms obtained from the loss function to update the parameters of the transformer architecture;
• and
• deploy the trained transformer architecture to an inference system.