Learning Student DNN Via Output Distribution

A computer implemented method for generating a trained DNN model for deployment as a classifier on a computer system, the method comprising:

• determining a plurality of DNN models to be included as sub-DNNs in an ensemble DNN model;
• assembling the ensemble DNN model using the sub-DNNs, thereby making each of plurality of sub-DNNs an ensemble member;
• training the ensemble DNN model;
• initializing a student DNN model;
• training the student DNN model, using the trained ensemble DNN model as a teacher DNN;
• and
• providing the student DNN model as a DNN classifier.

A DNN-based classifier deployed on a client device, the DNN-based classifier created according to a process comprising:

• (a) determining a first DNN model as a teacher DNN model;
• (b) initializing a second DNN model as a student DNN model;
• (c) receiving a set of un-labeled training data;
• (d) using a subset from the set of training data, determining a teacher output distribution for the teacher DNN model and a student output distribution for the student DNN model;
• (e) determining an evaluation of the student output distribution vs. the teacher output distribution;
• (e) based on the evaluation, determining whether the student output distribution and the teacher output distribution have achieved convergence:
• (i) if the student output distribution and the teacher output distribution are determined to have converged, then providing the student DNN model for deployment on the client device;
• and
• (ii) if the student output distribution and the teacher output distribution are determined not to have converged, then updating the student DNN model based on the determined evaluation and repeating steps (d) through (f).