# US 2022/0067280 A1 · Multi-token embedding and classifier for masked language models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0067280 A1
- Document ID: US20220067280A1
- Status: Published application
- Kind: A1
- Application number: 17/002,407
- Filed: 2020-08-25
- Published: 2022-03-03
- Earliest claimed: 2020-08-25
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Andy Wagner; Tiyasa Mitra; Marc Tremblay
- Web page: https://patentgrep.com/patent/US20220067280A1

## CPC

- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06F 40/30: Semantic analysis
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Embodiments of the present disclosure include systems and methods for training transformer models. In some embodiments, a set of input data are received. The input data comprises a plurality of tokens including masked tokens. The plurality of tokens in an embedding layer are processed. The embedding layer is coupled to a transformer layer. The plurality of tokens are processed in the transformer layer, which is coupled to a classifier layer. The plurality of tokens are processed in the classifier layer. The classifier layer is coupled to a loss layer. At least one of the embedding layer and the classifier layer combine masked tokens at a current position with tokens at one or more of a previous position and a subsequent position.

## Claims

### Claim 1

A computer system comprising:
one or more processors; and
a non-transitory computer-readable medium storing instructions that when executed by the one or more processors causes the one or more processors to:
receive a set of input data, the input data comprising a plurality of tokens, the plurality of tokens including masked tokens;
process the plurality of tokens in an embedding layer, the embedding layer being coupled to a transformer layer;
process the plurality of tokens in the transformer layer, the transformer layer being coupled to a classifier layer; and
process the plurality of tokens in the classifier layer, the classifier layer being coupled to a loss layer,
wherein one or more of the embedding layer and the classifier layer combine masked tokens at a current position with tokens at one or more of a previous position and a subsequent position.

### Claim 2 (depends on 1)

The computer system of claim 1 wherein the embedding layer combines the masked tokens at the current position with tokens at one or more previous positions and tokens at one or more subsequent positions.

### Claim 3 (depends on 1)

The computer system of claim 1 wherein the classifier layer combines the masked tokens at the current position with tokens at one or more previous positions and one or more subsequent positions.

### Claim 4 (depends on 1)

The computer system of claim 1 wherein the combining by the embedding layer comprises summing the masked tokens at the current position and the tokens at the one or more of the previous position and the subsequent position.

### Claim 5 (depends on 1)

The computer system of claim 1 wherein the combining by the classifier layer comprises concatenating the masked tokens at the current position and the tokens at the one or more of the previous position and the subsequent position.

### Claim 6 (depends on 1)

The computer system of claim 1 wherein the embedding layer comprises embedding tables to process in parallel masked tokens at the current position and the tokens at the one or more of the previous position and the subsequent position.

### Claim 7 (depends on 1)

The computer system of claim 1 wherein the classifier layer comprises gather modules to collect in parallel the masked tokens at the current position and the tokens at the one or more of the previous position and the subsequent position.

### Claim 8

A method comprising:
receiving a set of input data, the input data comprising a plurality of tokens, the plurality of tokens including masked tokens;
processing the plurality of tokens in an embedding layer, the embedding layer being coupled to a transformer layer;
processing the plurality of tokens in the transformer layer, the transformer layer being coupled to a classifier layer; and
processing the plurality of tokens in the classifier layer, the classifier layer being coupled to a loss layer,
wherein one or more of the embedding layer and the classifier layer combine masked tokens at a current position with tokens at one or more of a previous position and a subsequent position.

### Claim 9 (depends on 8)

The method of claim 8 wherein the embedding layer combines the masked tokens at the current position with tokens at one or more previous positions and tokens at one or more subsequent positions.

### Claim 10 (depends on 8)

The method of claim 8 wherein the classifier layer combines the masked tokens at the current position with tokens at one or more previous positions and one or more subsequent positions.

### Claim 11 (depends on 8)

The method of claim 8 wherein the combining by the embedding layer comprises summing the masked tokens at the current position and the tokens at the one or more of the previous position and the subsequent position.

### Claim 12 (depends on 8)

The method of claim 8 wherein the combining by the classifier layer comprises concatenating the masked tokens at the current position and the tokens at the one or more of the previous position and the subsequent position.

### Claim 13 (depends on 8)

The method of claim 8 wherein the embedding layer comprises embedding tables to process in parallel masked tokens at the current position and the tokens at the one or more of the previous position and the subsequent position.

### Claim 14 (depends on 8)

The method of claim 8 wherein the classifier layer comprises gather modules to collect in parallel the masked tokens at the current position and the tokens at the one or more of the previous position and the subsequent position.

### Claim 15

A non-transitory machine-readable medium storing a program executable by at least one processing unit, the programing comprising sets of instructions for:
receiving a set of input data, the input data comprising a plurality of tokens, the plurality of tokens including masked tokens;
processing the plurality of tokens in an embedding layer, the embedding layer being coupled to a transformer layer;
processing the plurality of tokens in the transformer layer, the transformer layer being coupled to a classifier layer; and
processing the plurality of tokens in the classifier layer, the classifier layer being coupled to a loss layer,
wherein one or more of the embedding layer and the classifier layer combine masked tokens at a current position with tokens at one or more of a previous position and a subsequent position.

### Claim 16 (depends on 15)

The non-transitory machine-readable medium of claim 15 wherein at least one of the embedding layer and the classifier layer combines the masked tokens at the current position with tokens at one or more previous positions and tokens at one or more subsequent positions.

### Claim 17 (depends on 15)

The non-transitory machine-readable medium of claim 15 wherein the combining by the embedding layer comprises summing the masked tokens at the current position and the tokens at the one or more of the previous position and the subsequent position.

### Claim 18 (depends on 15)

The non-transitory machine-readable medium of claim 15 wherein the combining by the classifier layer comprises concatenating the masked tokens at the current position and the tokens at the one or more of the previous position and the subsequent position.

### Claim 19 (depends on 15)

The non-transitory machine-readable medium of claim 15 wherein the embedding layer comprises embedding tables to process in parallel masked tokens at the current position and the tokens at the one or more of the previous position and the subsequent position.

### Claim 20 (depends on 15)

The non-transitory machine-readable medium of claim 15 wherein the classifier layer comprises gather modules to collect in parallel the masked tokens at the current position and the tokens at the one or more of the previous position and the subsequent position.
