# US 2020/0401929 A1 · Systems and Methods for Performing Knowledge Distillation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401929 A1 - Document ID: US20200401929A1 - Status: Published application - Kind: A1 - Application number: 16/445,651 - Filed: 2019-06-19 - Published: 2020-12-24 - Earliest claimed: 2019-06-19 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Thomas J. Duerig; Hongsheng Wang; Scott Alexander Rudkin - Web page: https://patentgrep.com/patent/US20200401929A1 ## CPC - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 20/00: Machine learning - G06N 3/045: Combinations of networks - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/09: Supervised learning - G06N 3/096: Transfer learning - G06N 5/02: Knowledge representation; Symbolic representation - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract The present disclosure is directed to methods and systems for knowledge distillation. Implementations of the disclosure can include executing the following actions using one or more computing devices: obtaining an initial training dataset including multiple training examples; determining sets of outputs by performing inference on the training examples with a group of pre-trained machine-learned models that have been trained to perform a respective task based on a respective pre-trained model training dataset; evaluating a performance of each pretrained machine-learned model based at least in part on the set of outputs generated by the pre-trained machine-learned model; determining for the set of outputs generated by each pre-trained machine-learned model, whether to include one or more outputs of the set of outputs in a distillation training dataset based at least in part on the respective performance of such pre-trained machine-learned model; and training a distilled machine-learned model using the distillation training dataset. ## Claims ### Claim 1 A computer-implemented method for performing knowledge distillation, the method comprising: obtaining, by one or more computing devices, an initial training dataset that comprises a set of training examples; obtaining, by the one or more computing devices, a plurality of sets of outputs respectively produced for the set of training examples by a plurality of pre-trained machine-learned models, each of the plurality of pre-trained machine-learned models having been previously trained to perform a respective task based on a respective pre-trained model training dataset; evaluating, by the one or more computing devices, a respective performance of each pre-trained machine-learned model based at least in part on the set of outputs generated by the pre-trained machine-learned model; determining, by the one or more computing devices, for the set of outputs generated by each pre-trained machine-learned model, whether to include one or more outputs of the set of outputs in a distillation training dataset based at least in part on the respective performance of such pre-trained machine-learned model; and training, by the one or more computing devices, a distilled machine-learned model using at least a portion of the distillation training dataset. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the plurality of pre-trained machine-learned models comprise pre-trained classifier models configured to infer one or more classification labels for each training example as an output. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein evaluating, by the one or more computing devices, the respective performance of each pre-trained machine-learned model based at least in part on the set of outputs generated by the pre-trained machine-learned model comprises evaluating, by the one or more computing devices, the respective performance of each pre-trained machine-learned model on a per-classification label basis. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, wherein determining, by the one or more computing devices for the set of outputs generated by each pre-trained machine-learned model, whether to include one or more outputs of the plurality set of outputs in the distillation training dataset comprises determining, by the one or more computing devices for each pre-trained machine-learned model and for each classification label, whether to include in the distillation training dataset all outputs of the set of outputs with the classification label. ### Claim 5 (depends on 3) The computer-implemented method of claim 3, wherein determining, by the one or more computing devices for the set of outputs generated by each pre-trained machine-learned model, whether to include one or more outputs of the plurality set of outputs in the distillation training dataset comprises, for each classification label: selecting, by the one or more computing devices, a highest-performing pre-trained machine-learned model for such classification label; and including, by the one or more computing devices, in the distillation training dataset all outputs of the set of outputs generated by the highest-performing pre-trained machine-learned model that have such classification label. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein evaluating, by the one or more computing devices, the respective performance of each pre-trained machine-learned model based at least in part on the set of outputs generated by the pre-trained machine-learned model comprises training, by the one or more computing devices using a validation dataset, one or more machine-learned trust models to evaluate the respective performance of each pre-trained machine-learned model. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein determining, by the one or more computing devices for the set of outputs generated by each pre-trained machine-learned model, whether to include one or more outputs of the set of outputs in the distillation training dataset based at least in part on the respective performance of such pre-trained machine-learned model comprises, for each pre-trained machine-learned model: providing, by the one or more computing devices, each respective training example as an input into at least one of the one or more trust models; and receiving, by the one or more computing devices, an output from the at least one of the one or more trust models that indicates whether the corresponding output generated by the pre-trained machine-learned model for the respective training example should be included in the distillation training dataset. ### Claim 8 (depends on 6) The computer-implemented method of claim 6, wherein the initial training dataset comprises a first portion that is labeled and a second portion that is not labeled, and wherein the first portion of the initial training dataset is used as the validation dataset. ### Claim 9 (depends on 6) The computer-implemented method of claim 6, wherein the one or more machine-learned trust models each comprise a neural network. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein evaluating, by the one or more computing devices, a respective performance of each pre-trained machine-learned model comprises: selecting, by the one or more computing devices, one or more expert models from the plurality of pre-trained machine-learned models by comparing a population statistic determined in part from the plurality of sets of outputs to the set of outputs determined by at least one of the pre-trained machine-learned models. ### Claim 11 (depends on 10) The computer-implemented method of claim 10 wherein selecting, by the one or more computing devices, one or more expert models comprises comparing the population statistic to each set of outputs determined respectively by each pre-trained machine-learned model. ### Claim 12 (depends on 10) The computer-implemented method of claim 10, wherein determining, by the one or more computing devices, whether to include one or more outputs of the plurality of sets of outputs in the distillation training dataset is further based at least in part on a weighting applied to each of the sets of outputs generated by the one or more expert models. ### Claim 13 (depends on 1) The computer-implemented method of claim 1, wherein at least 50% of the initial comprises unlabeled or weakly labeled training examples. ### Claim 14 (depends on 1) The computer-implemented method of claim 1, wherein obtaining the plurality of sets of outputs comprises respectively performing inference on the set of training examples with the plurality of pre-trained machine-learned models. ### Claim 15 A computing system configured to perform knowledge distillation, the computing system comprising: one or more processors; and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors cause the computing system to perform operations, the operations comprising: obtaining an initial training dataset that comprises a set of training examples and a plurality of sets of outputs for the set of training examples, the plurality of sets of outputs respectively produced by a plurality of pre-trained machine-learned models, each of the plurality of pre-trained machine-learned models having been previously trained to perform a respective task based on a respective pre-trained model training dataset; evaluating a respective performance of each pre-trained machine-learned model based at least in part on the set of outputs generated by the pre-trained machine-learned model; determining for the set of outputs generated by each pre-trained machine-learned model whether to include one or more outputs of the set of outputs in a distillation training dataset based at least in part on the respective performance of such pre-trained machine-learned model; and training a distilled machine-learned model using at least a portion of the distillation training dataset. ### Claim 16 (depends on 15) The computing system of claim 15, wherein evaluating the respective performance of each pre-trained machine-learned model based at least in part on the set of outputs generated by the pre-trained machine-learned model comprises training, using a validation dataset, one or more machine-learned trust models to evaluate the respective performance of each pre-trained machine-learned model. ### Claim 17 (depends on 15) The computing system of claim 15, wherein obtaining the plurality of sets of outputs comprises respectively performing inference on the set of training examples with the plurality of pre-trained machine-learned models. ### Claim 18 (depends on 15) The computing system of claim 15, wherein obtaining the plurality of sets of outputs comprises accessing the plurality of sets of outputs from a database that stores previously generated inferences for the plurality of pre-trained machine-learned models. ### Claim 19 (depends on 15) The computing system of claim 15, wherein evaluating a respective performance of each pre-trained machine-learned model comprises: selecting one or more expert models from the plurality of pre-trained machine-learned models by comparing a population statistic determined in part from the plurality of sets of outputs to the set of outputs determined by at least one pre-trained machine-learned model. ### Claim 20 One or more non-transitory computer-readable medium that store instructions for performing knowledge distillation that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising: obtaining an initial training dataset that comprises a set of training examples and a plurality of sets of outputs for the set of training examples, the plurality of sets of outputs respectively produced by a plurality of pre-trained machine-learned models, each of the plurality of pre-trained machine-learned models having been previously trained to perform a respective task based on a respective pre-trained model training dataset; evaluating a respective performance of each pre-trained machine-learned model based at least in part on the set of outputs generated by the pre-trained machine-learned model; determining for the set of outputs generated by each pre-trained machine-learned model whether to include one or more outputs of the set of outputs in a distillation training dataset based at least in part on the respective performance of such pre-trained machine-learned model; and training a distilled machine-learned model using at least a portion of the distillation training dataset.