Systems and Methods for Performing Knowledge Distillation

A computing system configured to perform knowledge distillation, the computing system comprising:

• one or more processors;
• and
• one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors cause the computing system to perform operations, the operations comprising:
• obtaining an initial training dataset that comprises a set of training examples and
• a plurality of sets of outputs for the set of training examples, the plurality of sets of outputs respectively produced by a plurality of pre-trained machine-learned models, each of the plurality of pre-trained machine-learned models having been previously trained to perform a respective task based on a respective pre-trained model training dataset;
• evaluating a respective performance of each pre-trained machine-learned model based at least in part on the set of outputs generated by the pre-trained machine-learned model;
• determining for the set of outputs generated by each pre-trained machine-learned model whether to include one or more outputs of the set of outputs in a distillation training dataset based at least in part on the respective performance of such pre-trained machine-learned model;
• and
• training a distilled machine-learned model using at least a portion of the distillation training dataset.

One or more non-transitory computer-readable medium that store instructions for performing knowledge distillation that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising:

• obtaining an initial training dataset that comprises a set of training examples and a plurality of sets of outputs for the set of training examples, the plurality of sets of outputs respectively produced by a plurality of pre-trained machine-learned models, each of the plurality of pre-trained machine-learned models having been previously trained to perform a respective task based on a respective pre-trained model training dataset;
• evaluating a respective performance of each pre-trained machine-learned model based at least in part on the set of outputs generated by the pre-trained machine-learned model;
• determining for the set of outputs generated by each pre-trained machine-learned model whether to include one or more outputs of the set of outputs in a distillation training dataset based at least in part on the respective performance of such pre-trained machine-learned model;
• and
• training a distilled machine-learned model using at least a portion of the distillation training dataset.