Compressing tokens based on positions for transformer models

A method comprising:

• receiving a set of input data for training a transformer model, the set of input data comprising a set of tokens and a set of position values, wherein each position value in the set of position values represents a position of a token in the set of tokens relative to other tokens in the set of tokens;
• identifying a first token in the set of tokens that is the same as a second token in the set of tokens;
• compressing the set of tokens by combining the position value representing the first token with the position value representing the second token;
• modifying the set of tokens by removing the first token from the set of tokens;
• generating a set of training data to comprise the modified set of tokens and the set of position values;
• and
• training the transformer model using the set of training data, wherein training the transformer model comprises:
• masking a set of selected tokens by replacing the selected tokens with a defined token value to generate masked tokens;
• generating predicted tokens for masked tokens;
• generating feedback based on differences between predicted tokens for masked tokens and actual values of masked tokens;
• and
• adjusting weights of the transformer model based on the feedback.

A non-transitory machine-readable medium storing a program executable by at least one processing unit of a computer system, the program comprising sets of instructions for:

• receiving a set of input data for training a transformer model, the set of input data comprising a set of tokens and a set of position values, wherein each position value in the set of position values represents a position of a token in the set of tokens relative to other tokens in the set of tokens;
• identifying a first token in the set of tokens that is the same as a second token in the set of tokens;
• compressing the set of tokens by combining the position value representing the first token with the position value representing the second token;
• modifying the set of tokens by removing the first token from the set of tokens;
• generating a set of training data to comprise the modified set of tokens and the set of position values;
• generating token embeddings, position embeddings, and token type embedding based on the training data;
• generating aggregate embeddings based on the token embeddings, the position embeddings, and the token type embedding;
• and
• training the transformer model using the aggregate embeddings set of training data, wherein training the transformer model comprises:
• masking a set of selected tokens by replacing the selected tokens with a defined token value to generate masked tokens;
• generating predicted tokens for masked tokens;
• generating feedback based on differences between predicted tokens for masked tokens and actual values of masked tokens;
• and
• adjusting weights of the transformer model based on the feedback.