Position masking for transformer models
AssigneeMicrosoft Technology Licensing, LLC
InventorsAndy Wagner, Tiyasa Mitra, Marc Tremblay
Life of this patent
Estimating the term…
Abstract
Embodiments of the present disclosure include systems and methods for training transformer models using position masking. In some embodiments, a set of data for training a transformer model is received. The set of data includes a sequence of tokens and a set of position values. Each position value in the set of position values represents a position of a token in the sequence of tokens relative to other tokens in the sequence of tokens. A subset of the set of position values in the set of data is selected. Each position value in the subset of the set of position values is replaced with a second defined value to form a second set of defined values. The transformer model is trained using the set of data.
Claims20
A system comprising:
- a set of processing units;
- and
- a non-transitory machine-readable medium storing instructions that when executed by at least one processing unit in the set of processing units cause the at least one processing unit to:
- receive a set of data for training a transformer model, the set of data comprising a sequence of tokens and a set of position values, wherein each position value in the set of position values represents a position of a token in the sequence of tokens relative to other tokens in the sequence of tokens;
- select a subset of the sequence of tokens in the set of data;
- replace each token in the subset of the sequence of tokens with a same first defined value to form a modified version of the sequence of tokens comprising a set of same first defined values;
- select a subset of the set of position values in the set of data;
- replace each position value in the subset of the set of position values with a same second defined value to form a modified version of the set of position values comprising a set of same second defined values, wherein the same second defined value represents a masked position value that is used when training the transformer model;
- and
- train the transformer model using the set of data that comprises the modified version of the sequence of tokens and the modified version of the set of position values.
A method comprising:
- receive a set of data for training a transformer model, the set of data comprising a sequence of tokens and a set of position values, wherein each position value in the set of position values represents a position of a token in the sequence of tokens relative to other tokens in the sequence of tokens;
- selecting a subset of the sequence of tokens in the set of data;
- replacing each token in the subset of the sequence of tokens with a same first defined value to form a modified version of the sequence of tokens comprising a set of same first defined values;
- selecting a subset of the set of position values in the set of data;
- replacing each position value in the subset of the set of position values with a same second defined value to form a modified version of the set of position values comprising a set of same second defined values, wherein the same second defined value represents a masked position value that is used when training the transformer model;
- and
- training the transformer model using the set of data that comprises the modified version of the sequence of tokens and the modified version of the set of position values.
A non-transitory machine-readable medium storing a program executable by at least one processing unit of a computer system, the program comprising sets of instructions for:
- receive a set of data for training a transformer model, the set of data comprising a sequence of tokens and a set of position values, wherein each position value in the set of position values represents a position of a token in the sequence of tokens relative to other tokens in the sequence of tokens;
- selecting a subset of the sequence of tokens in the set of data;
- replacing each token in the subset of the sequence of tokens with a same first defined value to form a modified version of the sequence of tokens comprising a set of same first defined values;
- selecting a subset of the set of position values in the set of data;
- replacing each position value in the subset of the set of position values with a same second defined value to form a modified version of the set of position values comprising a set of same second defined values, wherein the same second defined value represents a masked position value that is used when training the transformer model;
- and
- training the transformer model using the set of data that comprises the modified version of the sequence of tokens and the modified version of the set of position values.
Dependencies are inferred from the claim text.