Filtering data for knowledge distillation

A system comprising:

• one or more computers;
• and
• one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
• obtaining a training dataset comprising a plurality of training inputs;
• generating a filtered training dataset comprising a subset of the plurality of training inputs, comprising, for each of the one or more training inputs:
• processing the training input using a teacher neural network to generate a respective teacher output for the task,
• determining an uncertainty measure for the training input that represents an uncertainty of the teacher neural network in generating the respective teacher output;
• and
• determining whether to filter out the training input based on the uncertainty measure;
• and
• training the student neural network using the filtered training dataset.

One or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• obtaining a training dataset comprising a plurality of training inputs;
• generating a filtered training dataset comprising a subset of the plurality of training inputs, comprising, for each of the one or more training inputs:
• processing the training input using a teacher neural network to generate a respective teacher output for the task,
• determining an uncertainty measure for the training input that represents an uncertainty of the teacher neural network in generating the respective teacher output;
• and
• determining whether to filter out the training input based on the uncertainty measure;
• and
• training the student neural network using the filtered training dataset.