Scalable knowledge distillation techniques for machine learning

A method implemented in a data processing system for distilling a student model from a teacher model, the method comprising:

• dividing training data into a plurality of batches of samples;
• and
• distilling a student model from a teacher model using an iterative knowledge distillation process by:
• instantiating an instance of the teacher model in a memory of the data processing system;
• instantiating an instance of the student model in the memory;
• obtaining a respective batch of training data from the plurality of batches of samples in the memory;
• training the teacher model using each of the samples in the respective batch of the training data;
• training the student model using each of the samples in the respective batch of the training data;
• evaluating performance of the student model compared with the performance of the teacher model;
• and
• providing feedback to student model to adjust behavior of the student model based on the performance of the student model.

A machine-readable medium on which are stored instructions that, when executed, cause a processor of a programmable device to perform operations of:

• dividing training data into a plurality of batches of samples;
• and
• distilling a student model from a teacher model using an iterative knowledge distillation process by:
• instantiating an instance of the teacher model in a memory of the programmable device;
• instantiating an instance of the student model in the memory;
• obtaining a respective batch of training data from the plurality of batches of samples in the memory;
• training the teacher model using each of the samples in the respective batch of the training data;
• training the student model using each of the samples in the respective batch of the training data;
• evaluating performance of the student model compared with the performance of the teacher model;
• and
• providing feedback to student model to adjust behavior of the student model based on the performance of the student model.