Systems and Methods for Pretraining Models for Diverse Downstream Tasks
AssigneeGoogle Inc.
InventorsYi Tay, Mostafa Dehghani
Life of this patent
Estimating the term…
Loading patent…
AssigneeGoogle Inc.
InventorsYi Tay, Mostafa Dehghani
Estimating the term…
An example method for pretraining a machine-learned model is provided. The example method includes obtaining a plurality of different combinations of configuration parameters of a pretraining objective framework. The example method includes generating, using the pretraining objective framework, a plurality of corrupted training examples from one or more training examples, wherein the plurality of corrupted training examples are respectively generated according to the plurality of different combinations. The example method includes inputting the plurality of corrupted training examples into the machine-learned model, wherein the machine-learned model is configured to generate uncorrupted subportions corresponding to corrupted subportions of the corrupted training examples. The example method includes obtaining, from the machine-learned model, a plurality of outputs respectively generated by the machine-learned model based on the plurality of corrupted training examples. The example method includes updating one or more parameters of the machine-learned model based on an evaluation of the plurality of outputs.
A computer-implemented method for pretraining a machine-learned model with diversified objectives, comprising:
A non-transitory, computer-readable medium storing instructions that are executable by one or more processors to cause a computing system to perform operations, the operations comprising:
A computing system, comprising:
21-27. (canceled)
Dependencies are inferred from the claim text.