US 2022/0067280 A1 · MULTI-TOKEN EMBEDDING AND CLASSIFIER FOR MASKED LANGUAGE MODELS

Abstract

Embodiments of the present disclosure include systems and methods for training transformer models. In some embodiments, a set of input data are received. The input data comprises a plurality of tokens including masked tokens. The plurality of tokens in an embedding layer are processed. The embedding layer is coupled to a transformer layer. The plurality of tokens are processed in the transformer layer, which is coupled to a classifier layer. The plurality of tokens are processed in the classifier layer. The classifier layer is coupled to a loss layer. At least one of the embedding layer and the classifier layer combine masked tokens at a current position with tokens at one or more of a previous position and a subsequent position.

Claims20

Dependent claims

A computer system comprising:

one or more processors;
and
a non-transitory computer-readable medium storing instructions that when executed by the one or more processors causes the one or more processors to:
receive a set of input data, the input data comprising a plurality of tokens, the plurality of tokens including masked tokens;
process the plurality of tokens in an embedding layer, the embedding layer being coupled to a transformer layer;
process the plurality of tokens in the transformer layer, the transformer layer being coupled to a classifier layer;
and
process the plurality of tokens in the classifier layer, the classifier layer being coupled to a loss layer,
wherein one or more of the embedding layer and the classifier layer combine masked tokens at a current position with tokens at one or more of a previous position and a subsequent position.

6 more elements

Dependencies are inferred from the claim text.

Multi-token embedding and classifier for masked language models

Life of this patent

Multi-token embedding and classifier for masked language models

Life of this patent

Abstract

Claims20