# US 2022/0027576 A1 · Determining position values for transformer models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0027576 A1
- Document ID: US20220027576A1
- Status: Published application
- Kind: A1
- Application number: 16/935,072
- Filed: 2020-07-21
- Published: 2022-01-27
- Earliest claimed: 2020-07-21
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Andy WAGNER; Tiyasa MITRA; Marc TREMBLAY
- Web page: https://patentgrep.com/patent/US20220027576A1

## CPC

- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06F 40/216: using statistical methods
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0499: Feedforward networks
- G06N 3/09: Supervised learning
- G06N 5/04: Inference or reasoning models
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Embodiments of the present disclosure include systems and methods for determining position values for training data that is used to train transformer models. In some embodiments, a set of input data for training a transformer model is received. The set of input data comprises a set of tokens. Based on an offset value, a set of successive position values for the set of tokens is determined. Each position value in the set of successive position values represents a position of a token in the set of tokens relative to other tokens in the set of tokens. A set of training data is generated to comprise the set of tokens and the set of successive position values. The transformer model is trained using the set of training data.

## Claims

### Claim 1

A system comprising:
a set of processing units; and
a non-transitory machine-readable medium storing instructions that when executed by at least one processing unit in the set of processing units cause the at least one processing unit to:
receive a set of input data for training a transformer model, the set of input data comprising a set of tokens;
based on an offset value, determine a set of successive position values for the set of tokens, wherein each position value in the set of successive position values represents a position of a token in the set of tokens relative to other tokens in the set of tokens;
generate a set of training data to comprise the set of tokens and the set of successive position values; and
train the transformer model using the set of training data.

### Claim 2 (depends on 1)

The system of claim 1, wherein the set of tokens is a first set of tokens, wherein the set of input data further comprises a second set of tokens, wherein the offset value is a first offset value, wherein the set of successive position values is a first set of successive position values,
wherein the instructions further cause the at least one processing unit to, based on a second offset value, determine a second set of successive position values for the second set of tokens, wherein each position value in the second set of successive position values represents a position of a token in the second set of tokens relative to other tokens in the second set of tokens,
wherein generating the set of training data comprises generate the set of training data to comprise the first set of tokens, the first set of successive position values, the second set of tokens, and the second set of successive position value.

### Claim 3 (depends on 2)

The system of claim 2, wherein the first offset value and the second offset value are the same.

### Claim 4 (depends on 3)

The system of claim 3, wherein the first offset value is the value zero.

### Claim 5 (depends on 2)

The system of claim 2, wherein the first offset value and the second offset value are different.

### Claim 6 (depends on 2)

The system of claim 2, wherein a position value in the first set of successive position values has the same value as a position value in the second set of successive position values.

### Claim 7 (depends on 1)

The system of claim 1, wherein the set of training data is a first set of training data, wherein the offset value is a first offset value, wherein the set of successive position values is a first set of successive position values, wherein the set of training data is a first set of training data, wherein the instructions further cause the at least one processing unit to:
based on a second offset value, determine a second set of successive position values for the set of tokens, wherein each position value in the second set of successive position values represents a position of a token in the set of tokens relative to other tokens in the set of token;
generate a second set of training data to comprise the set of tokens and the second set of successive position values; and
train the transformer model using the second set of training data.

### Claim 8 (depends on 7)

The system of claim 7, wherein a difference between the first offset value and the second offset value is a defined difference value.

### Claim 9 (depends on 1)

The system of claim 1, wherein the instructions further cause the at least one processing unit to determine the offset value by randomly selecting the offset value from a range of candidate offset values.

### Claim 10 (depends on 1)

The system of claim 1, wherein the transformer model is configured to train on training data comprising a sequence of tokens that is less than a defined maximum number of tokens, wherein each position value in the set of successive position values is less than the defined maximum number of tokens.

### Claim 11

A method comprising:
receiving a set of input data for training a transformer model, the set of input data comprising a set of tokens;
based on an offset value, determining a set of successive position values for the set of tokens, wherein each position value in the set of successive position values represents a position of a token in the set of tokens relative to other tokens in the set of tokens;
generating a set of training data to comprise the set of tokens and the set of successive position values; and
training the transformer model using the set of training data.

### Claim 12 (depends on 11)

The method of claim 11, wherein the set of tokens is a first set of tokens, wherein the set of input data further comprises a second set of tokens, wherein the offset value is a first offset value, wherein the set of successive position values is a first set of successive position values,
wherein the method further comprises, based on a second offset value, determine a second set of successive position values for the second set of tokens, wherein each position value in the second set of successive position values represents a position of a token in the second set of tokens relative to other tokens in the second set of tokens,
wherein generating the set of training data comprises generate the set of training data to comprise the first set of tokens, the first set of successive position values, the second set of tokens, and the second set of successive position value.

### Claim 13 (depends on 11)

The method of claim 11, wherein the set of training data is a first set of training data, wherein the offset value is a first offset value, wherein the set of successive position values is a first set of successive position values, wherein the set of training data is a first set of training data, wherein the method further comprises:
based on a second offset value, determining a second set of successive position values for the set of tokens, wherein each position value in the second set of successive position values represents a position of a token in the set of tokens relative to other tokens in the set of token;
generating a second set of training data to comprise the set of tokens and the second set of successive position values; and
training the transformer model using the second set of training data.

### Claim 14 (depends on 11)

The method of claim 11 further comprising determining the offset value by randomly selecting the offset value from a range of candidate offset values.

### Claim 15 (depends on 11)

The method of claim 11, wherein the transformer model is configured to train on training data comprising a sequence of tokens that is less than a defined maximum number of tokens, wherein each position value in the set of successive position values is less than the defined maximum number of tokens.

### Claim 16

A non-transitory machine-readable medium storing a program executable by at least one processing unit of a computer system, the program comprising sets of instructions for:
receiving a set of input data for training a transformer model, the set of input data comprising a set of tokens;
based on an offset value, determining a set of successive position values for the set of tokens, wherein each position value in the set of successive position values represents a position of a token in the set of tokens relative to other tokens in the set of tokens;
generating a set of training data to comprise the set of tokens and the set of successive position values; and
training the transformer model using the set of training data.

### Claim 17 (depends on 16)

The non-transitory machine-readable medium of claim 16, wherein the set of tokens is a first set of tokens, wherein the set of input data further comprises a second set of tokens, wherein the offset value is a first offset value, wherein the set of successive position values is a first set of successive position values,
wherein the program further comprises a set of instructions for, based on a second offset value, determining a second set of successive position values for the second set of tokens, wherein each position value in the second set of successive position values represents a position of a token in the second set of tokens relative to other tokens in the second set of tokens,
wherein generating the set of training data comprises generate the set of training data to comprise the first set of tokens, the first set of successive position values, the second set of tokens, and the second set of successive position value.

### Claim 18 (depends on 16)

The non-transitory machine-readable medium of claim 16, wherein the set of training data is a first set of training data, wherein the offset value is a first offset value, wherein the set of successive position values is a first set of successive position values, wherein the set of training data is a first set of training data, wherein the program further comprises sets of instructions for:
based on a second offset value, determining a second set of successive position values for the set of tokens, wherein each position value in the second set of successive position values represents a position of a token in the set of tokens relative to other tokens in the set of token;
generating a second set of training data to comprise the set of tokens and the second set of successive position values; and
training the transformer model using the second set of training data.

### Claim 19 (depends on 16)

The non-transitory machine-readable medium of claim 16, wherein the program further comprises a set of instructions for determining the offset value by randomly selecting the offset value from a range of candidate offset values.

### Claim 20 (depends on 16)

The non-transitory machine-readable medium of claim 16, wherein the transformer model is configured to train on training data comprising a sequence of tokens that is less than a defined maximum number of tokens, wherein each position value in the set of successive position values is less than the defined maximum number of tokens.
