Determining position values for transformer models
AssigneeMicrosoft Technology Licensing, LLC
InventorsAndy WAGNER, Tiyasa MITRA, Marc TREMBLAY
Life of this patent
Estimating the term…
Loading patent…
AssigneeMicrosoft Technology Licensing, LLC
InventorsAndy WAGNER, Tiyasa MITRA, Marc TREMBLAY
Estimating the term…
Embodiments of the present disclosure include systems and methods for determining position values for training data that is used to train transformer models. In some embodiments, a set of input data for training a transformer model is received. The set of input data comprises a set of tokens. Based on an offset value, a set of successive position values for the set of tokens is determined. Each position value in the set of successive position values represents a position of a token in the set of tokens relative to other tokens in the set of tokens. A set of training data is generated to comprise the set of tokens and the set of successive position values. The transformer model is trained using the set of training data.
A system comprising:
A method comprising:
A non-transitory machine-readable medium storing a program executable by at least one processing unit of a computer system, the program comprising sets of instructions for:
Dependencies are inferred from the claim text.