# US 2025/0156756 A1 · Systems and Methods for Pretraining Models for Diverse Downstream Tasks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0156756 A1
- Document ID: US20250156756A1
- Status: Published application
- Kind: A1
- Application number: 18/835,666
- Filed: 2022-12-30
- Published: 2025-05-15
- Earliest claimed: 2022-02-02
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Yi Tay; Mostafa Dehghani
- Web page: https://patentgrep.com/patent/US20250156756A1

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 20/00: Machine learning
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F 40/30: Semantic analysis
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/096: Transfer learning
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0499: Feedforward networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An example method for pretraining a machine-learned model is provided. The example method includes obtaining a plurality of different combinations of configuration parameters of a pretraining objective framework. The example method includes generating, using the pretraining objective framework, a plurality of corrupted training examples from one or more training examples, wherein the plurality of corrupted training examples are respectively generated according to the plurality of different combinations. The example method includes inputting the plurality of corrupted training examples into the machine-learned model, wherein the machine-learned model is configured to generate uncorrupted subportions corresponding to corrupted subportions of the corrupted training examples. The example method includes obtaining, from the machine-learned model, a plurality of outputs respectively generated by the machine-learned model based on the plurality of corrupted training examples. The example method includes updating one or more parameters of the machine-learned model based on an evaluation of the plurality of outputs.

## Claims

### Claim 1

A computer-implemented method for pretraining a machine-learned model with diversified objectives, comprising:
obtaining, by a computing system comprising one or more processors, a plurality of different combinations of configuration parameters of a pretraining objective framework;
generating, by the computing system and using the pretraining objective framework, a plurality of corrupted training examples from one or more training examples, wherein the plurality of corrupted training examples are respectively generated according to the plurality of different combinations of configuration parameters;
inputting, by the computing system, the plurality of corrupted training examples into the machine-learned model, wherein the machine-learned model is configured to generate uncorrupted subportions corresponding to corrupted subportions of the corrupted training examples;
obtaining, by the computing system and from the machine-learned model, a plurality of outputs respectively generated by the machine-learned model based on the plurality of corrupted training examples; and
updating, by the computing system, one or more parameters of the machine-learned model based on an evaluation of the plurality of outputs.

### Claim 2 (depends on 1)

The method of claim 1, wherein the configuration parameters comprise two or more different parameters of: a subportion length parameter, a subportion quantity parameter, or a corruption rate parameter.

### Claim 3 (depends on 1)

The method of claim 1, wherein the plurality of different combinations of configuration parameters comprise:
a distributed configuration configured for generating a plurality of corrupted subportions distributed over a training example; and
a sequential configuration configured for generating a corrupted subportion corresponding to a terminus of the training example.

### Claim 4 (depends on 1)

The method of claim 1, wherein the plurality of different combinations of configuration parameters comprise:
a first distributed configuration configured for generating a first plurality of corrupted subportions distributed over a training example;
a second distributed configuration configured for generating a second plurality of corrupted subportions distributed over the training example, wherein the second distributed configuration is configured to cause greater corruption of the training example than the first distributed configuration; and
a sequential configuration configured for generating a corrupted subportion corresponding to a terminus of the training example.

### Claim 5 (depends on 4)

The method of claim 4, wherein, as compared to the first distributed configuration, the second distributed configuration comprises at least one of:
a subportion length parameter corresponding to a longer subportion length; or
a corruption rate parameter corresponding to a greater rate of corruption.

### Claim 6 (depends on 4)

The method of claim 4, wherein the sequential configuration corresponds to a prefix-based language modeling objective.

### Claim 7 (depends on 1)

The method of claim 1, wherein the plurality of different combinations of configuration parameters comprises:
a first plurality of distributed configurations that are respectively associated with subportion length parameters indicating subportion lengths of less than about 12 tokens;
a second plurality of distributed configurations that are respectively associated with at least one of:
subportion length parameters indicating subportion lengths of greater than about 12 tokens; or
corruption rate parameters indicating a corruption rate of greater than about 30%.

### Claim 8 (depends on 7)

The method of claim 7, wherein the first plurality of distributed configurations are respectively associated with subportion length parameters indicating subportion lengths of less than about 10 tokens.

### Claim 9 (depends on 8)

The method of claim 8, wherein the second plurality of distributed configurations are respectively associated with subportion length parameters indicating subportion lengths of greater than about 12 tokens.

### Claim 10 (depends on 9)

The method of claim 9, wherein the second plurality of distributed configurations are respectively associated with subportion length parameters indicating subportion lengths of greater than about 30 tokens.

### Claim 11 (depends on 9)

The method of claim 9, wherein the second plurality of distributed configurations are respectively associated with corruption rate parameters indicating a corruption rate of greater than about 30%.

### Claim 12 (depends on 11)

The method of claim 11, wherein the second plurality of distributed configurations are respectively associated with corruption rate parameters indicating a corruption rate of at least about 50%.

### Claim 13 (depends on 1)

The method of claim 1, wherein generating a plurality of corrupted training examples from the one or more training examples comprises:
for a respective training example of the one or more training examples, the respective training example comprising a respective sequence of data tokens:
determining, by the computing system, one or more selected subportions of the respective sequence of data tokens; and
replacing, by the computing system, the one or more selected subportions with a replacement token.

### Claim 14 (depends on 1)

The method of claim 1, comprising:
inputting, by the computing system and with a respective corrupted training example of the plurality of corrupted training examples, a mode-switching token corresponding to at least one configuration of the plurality of different combinations of configuration parameters, the at least one configuration used to corrupt the respective corrupted training example.

### Claim 15 (depends on 14)

The method of claim 14, wherein the mode-switching token triggers downstream behavior of the machine-learned model corresponding to tasks prioritized by the at least one configuration.

### Claim 16 (depends on 1)

The method of claim 1, wherein at least one of the corruption parameters is a probabilistic parameter.

### Claim 17 (depends on 16)

The method of claim 16, wherein the probabilistic parameter is the subportion length parameter characterizing a distribution from which a selected subportion length is sampled.

### Claim 18 (depends on 16)

The method of claim 16, wherein the probabilistic parameter is the corruption rate parameter characterizing a rate at which one or more selected subportions of a training example are corrupted.

### Claim 19

A non-transitory, computer-readable medium storing instructions that are executable by one or more processors to cause a computing system to perform operations, the operations comprising:
obtaining a plurality of different combinations of configuration parameters of a pretraining objective framework;
generating, using the pretraining objective framework, a plurality of corrupted training examples from one or more training examples, wherein the plurality of corrupted training examples are respectively generated according to the plurality of different combinations of configuration parameters;
inputting the plurality of corrupted training examples into the machine-learned model, wherein the machine-learned model is configured to generate uncorrupted subportions corresponding to corrupted subportions of the corrupted training examples;
obtaining, from the machine-learned model, a plurality of outputs respectively generated by the machine-learned model based on the plurality of corrupted training examples; and
updating one or more parameters of the machine-learned model based on an evaluation of the plurality of outputs.

### Claim 20

A computing system, comprising:
one or more processors; and
a non-transitory, computer-readable medium storing instructions that are executable by the one or more processors to cause the computing system to perform operations, the operations comprising:
obtaining a plurality of different combinations of configuration parameters of a pretraining objective framework;
generating, using the pretraining objective framework, a plurality of corrupted training examples from one or more training examples, wherein the plurality of corrupted training examples are respectively generated according to the plurality of different combinations of configuration parameters;
inputting the plurality of corrupted training examples into the machine-learned model, wherein the machine-learned model is configured to generate uncorrupted subportions corresponding to corrupted subportions of the corrupted training examples;
obtaining, from the machine-learned model, a plurality of outputs respectively generated by the machine-learned model based on the plurality of corrupted training examples; and
updating one or more parameters of the machine-learned model based on an evaluation of the plurality of outputs.

### Claim at position 21

21-27. (canceled)
