Efficient transformer language models with disentangled attention and multi-step decoding

A computing system configured to improve pre-training convergence while encoding data with a transformer, the computing system comprising:

• one or more processors;
• and
• one or more computer-readable hardware storage devices that store computer executable instructions that are executable by the one or more processors to cause the computer system to at least:
• identify a plurality of tokens to be encoded from a sequence;
• obtain a transformer that includes an encoder with a plurality of encoding layers;
• embed the plurality of tokens to generate input data;
• apply the input data to the encoder;
• and
• apply output of a final encoding layer as additional input to the final encoding layer for one or more iterations in order to generate new output from the final encoding layer.

A method of using encoding data with a transformer that is configured to account for position bias and improve pre-training convergence, the method including:

• identifying a plurality of tokens to be encoded from a sequence;
• obtaining a transformer that includes an encoder, comprising a plurality encoding layers within the encoder wherein each of the plurality of encoding layers includes a self-attention sub-layer and a feed forward sub-layer;
• embedding the plurality of tokens to generate input data;
• applying the input data to the encoder;
• generating and applying an attention score at the self-attention sub-layer, by at least disentangling position bias embedding from content embedding associated with the plurality of the tokens, the attention score comprising a summation of:
• a first attention score component comprising a product of a content embedding of a first token in a token pair, a first learn-able content parameter, a second learn-able content parameter, and a content embedding of a second token in a token pair;
• a second attention score component comprising a product of the content embedding of the first token in the token pair, the first learn-able content parameter, a first learnable position parameter, and a relative position embedding of the first and second tokens in a token pair;
• and
• a third attention score component comprising a product of the relative position embedding of the first and second tokens in the token pair, a second learn-able position parameter, the second learn-able content parameter, and the second content embedding of a second token in a token pair;
• and
• applying output of a final encoding layer as additional input to the final encoding layer in order to generate new output from the final encoding layer.