Pretraining Already-Pretrained Models for Diverse Downstream Tasks

One or more non-transitory, computer-readable media storing:

• a pretrained machine-learned model having parameters that were obtained using at least two stages of pretraining, a first stage of pretraining using a pretraining dataset, and a second stage of the pretraining comprising:
• further pretraining the pretrained machine-learned model by:
• generating using a pretraining objective framework, a plurality of corrupted training examples from one or more training examples obtained from the pretraining dataset,
• wherein the plurality of corrupted training examples comprise:
• a first set of one or more training examples corrupted according to a first set of configuration parameters of the pretraining objective framework;
• and
• a second set of one or more training examples corrupted according to a second set of configuration parameters of the pretraining objective framework;
• inputting the plurality of corrupted training examples into the pretrained machine-learned model, wherein the pretrained machine-learned model is configured to generate uncorrupted subportions corresponding to corrupted subportions of the plurality of corrupted training examples;
• obtaining, from the pretrained machine-learned model, a plurality of outputs respectively generated by the pretrained machine-learned model based on the plurality of corrupted training examples;
• and
• updating one or more parameters of the pretrained machine-learned model based on an evaluation of the plurality of outputs.

A computing system, comprising:

• one or more processors;
• and
• one or more non-transitory, computer-readable media storing:
• a pretrained machine-learned model having parameters that were obtained using at least two stages of pretraining, a first stage of pretraining using a pretraining dataset, and a second stage of the pretraining comprising:
• further pretraining the pretrained machine-learned model by:
• generating, using a pretraining objective framework, a plurality of corrupted training examples from one or more training examples obtained from the pretraining dataset,
• wherein the plurality of corrupted training examples comprise:
• a first set of one or more training examples corrupted according to a first set of configuration parameters of the pretraining objective framework;
• and
• a second set of one or more training examples corrupted according to a second set of configuration parameters of the pretraining objective framework;
• inputting the plurality of corrupted training examples into the pretrained machine-learned model, wherein the pretrained machine-learned model is configured to generate uncorrupted subportions corresponding to corrupted subportions of the plurality of corrupted training examples;
• obtaining, from the pretrained machine-learned model, a plurality of outputs respectively generated by the pretrained machine-learned model based on the plurality of corrupted training examples;
• and
• updating one or more parameters of the pretrained machine-learned model based on an evaluation of the plurality of outputs;
• instructions that are executable to cause the one or more processors to perform operations, the operations comprising:
• receiving one or more inputs for processing using the pretrained machine-learned model;
• and
• generating, by processing the one or more inputs using the pretrained machine-learned model, one or more outputs.