Selective knowledge distillation

A system comprising one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

• providing, to a teacher model in a knowledge distillation training system, first data representing an image to cause the teacher model to generate teacher output data, the teacher output data comprising (i) features of the teacher model and (ii) predictions that indicate whether the image depicts an object of interest;
• providing, to a student model in the knowledge distillation training system, second data representing the image to cause the student model to generate student output data, the student output data comprising (i) features of the student model and (ii) predictions that indicate whether the image depicts an object of interest;
• determining whether an accuracy of the teacher output data satisfies an accuracy threshold;
• and
• in response to determining that the accuracy of the teacher output data does not satisfy the accuracy threshold:
• determining to skip updating the student model using a feature-based distillation loss for the teacher model, the feature-based distillation loss determined based on the features of the student and teacher models;
• determining to skip updating the student model using a prediction-based distillation loss for the teacher model, the prediction-based distillation loss determined based on the predictions of the student and teacher models;
• and
• updating the student model using the student output data and ground truth data for the image.

One or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

• providing, to a teacher model in a knowledge distillation training system, first data representing an image to cause the teacher model to generate teacher output data, the teacher output data comprising (i) features of the teacher model and (ii) predictions that indicate whether the image depicts an object of interest;
• providing, to a student model in the knowledge distillation training system, second data representing the image to cause the student model to generate student output data, the student output data comprising (i) features of the student model and (ii) predictions that indicate whether the image depicts an object of interest;
• determining whether an accuracy of the teacher output data satisfies an accuracy threshold;
• and
• in response to determining that the accuracy of the teacher output data does not satisfy the accuracy threshold:
• determining to skip updating the student model using a feature-based distillation loss for the teacher model, the feature-based distillation loss determined based on the features of the student and teacher models;
• and
• updating the student model using the student output data and ground truth data for the image.