Token Packing for Sequence Models
AssigneeMicrosoft Technology Licensing, LLC
InventorsAndy WAGNER, Tiyasa MITRA, Marc TREMBLAY
Life of this patent
Estimating the term…
Abstract
Embodiments of the present disclosure include systems and methods for packing tokens to train sequence models. In some embodiments, a plurality of datasets for training a sequence model is received. Each dataset in the plurality of datasets includes a sequence of correlated tokens. A set of training data is generated that includes a subset of a sequence of tokens from a first dataset in the plurality of datasets and a subset of a sequence of tokens from a second, different dataset in the plurality of datasets. The sequence model is trained using the set of training data.
Claims20
A method comprising:
- receiving a set of input data for training a sequence model, the input data comprising a sequence of tokens;
- grouping the sequence of tokens into a first group of tokens and a second group of tokens;
- generating a set of training data by:
- packing a data structured with the first group of tokens and the second group of tokens and
- repeatedly packing the data structure with copies of at least a portion of the first and second groups of tokens until the data structure is filled up;
- and
- training the sequence model using the set of training data.
A system comprising:
- a set of processing units;
- and
- a non-transitory machine-readable medium storing instructions that when executed by at least one processing unit in the set of processing units cause the at least one processing unit to:
- receive a set of input data for training a sequence model, the input data comprising a sequence of tokens;
- group the sequence of tokens into a first group of tokens and a second group of tokens;
- generate a set of training data comprising a data structure by:
- packing the data structured with the first group of tokens and the second group of tokens and
- repeatedly packing the data structure with copies of at least a portion of the first group of tokens and copies of at least a portion of the second group of tokens until the data structure is filled up;
- and
- train the sequence model using the set of training data.
A non-transitory machine-readable medium storing a program executable by at least one processing unit of a device, the program comprising sets of instructions for:
- receiving a set of input data for training a sequence model, the input data comprising a sequence of tokens;
- grouping the sequence of tokens into a first group of tokens and a second group of tokens;
- generating a set of training data by:
- packing a data structured with the first group of tokens and the second group of tokens and
- repeatedly packing the data structure with copies of at least a portion of the first and second groups of tokens until the data structure is filled up;
- and
- training the sequence model using the set of training data.
Dependencies are inferred from the claim text.