Self-supervised quantization-aware knowledge distillation for neural networks

A system comprising:

• processing circuitry;
• non-transitory computer readable media;
• and
• instructions that, when executed by the processing circuitry, configure the processing circuitry to:
• obtain, by the processing circuitry, input data for training an artificial intelligence model (AI model);
• provide, by the processing circuitry, the input data into a training framework having both a teacher layer providing target outputs and a student layer configured to learn to approximate the target outputs;
• train the AI model using the training framework, wherein to train the AI model includes the instructions, when executed, to further configure the processing circuitry to:
• define, using the training framework, a quantizer that includes a clipping function and a rounding function to convert full-precision data derived from the input data to quantized data;
• determine, using the training framework, a first output distribution for the input data from the teacher layer of the training framework using the full-precision data;
• determine, using the training framework, a second output distribution for the input data from the student layer of the training framework using the quantized data;
• calculate, using the training framework, a training loss between the first output distribution from the teacher layer and the second output distribution from the student layer;
• and
• modify, using the training framework, training parameters of a quantized network of the student layer to reduce discretization error and prediction discrepancy between the quantized network of the student layer and a full-precision network of the teacher layer;
• generate, by the processing circuitry, new predictive output from the AI model using the quantized network of the student layer trained on the input data using the training framework;
• and
• output, by the processing circuitry, the predictive output.

Computer-readable storage media comprising instructions that, when executed, configure processing circuitry to:

• obtain input data for training an artificial intelligence model (AI model);
• provide the input data into a training framework having both a teacher layer providing target outputs and a student layer configured to learn to approximate the target outputs;
• train the AI model using the training framework, wherein to train the AI model includes the instructions, when executed, to further configure the processing circuitry to:
• define a quantizer that includes a clipping function and a rounding function to convert full-precision data derived from the input data to quantized data;
• determine a first output distribution for the input data from the teacher layer of the training framework using the full-precision data;
• determine a second output distribution for the input data from the student layer of the training framework using the quantized data;
• calculate a training loss between the first output distribution from the teacher layer and the second output distribution from the student layer;
• and
• modify training parameters of a quantized network of the student layer to reduce discretization error and prediction discrepancy between the quantized network of the student layer and a full-precision network of the teacher layer;
• generate new predictive output from the AI model using the quantized network of the student layer trained on the input data using the training framework;
• and
• output the predictive output.