US 10,650,328 B2 · Training distilled machine learning models

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a distilled machine learning model. One of the methods includes training a cumbersome machine learning model, wherein the cumbersome machine learning model is configured to receive an input and generate a respective score for each of a plurality of classes; and training a distilled machine learning model on a plurality of training inputs, wherein the distilled machine learning model is also configured to receive inputs and generate scores for the plurality of classes, comprising: processing each training input using the cumbersome machine learning model to generate a cumbersome target soft output for the training input; and training the distilled machine learning model to, for each of the training inputs, generate a soft output that matches the cumbersome target soft output for the training input.

A method performed by one or more computers, the method comprising:

obtaining data defining a cumbersome neural network having a plurality of cumbersome model parameters, wherein:
the cumbersome neural network is configured to process an input to generate a respective score for each of a plurality of classes;
and
the cumbersome neural network has been trained to determine trained values of the cumbersome model parameters;
and
training a distilled neural network having a plurality of distilled model parameters on a set of multiple training inputs to determine trained values of the distilled model parameters, wherein the distilled neural network has fewer parameters than the cumbersome neural network, wherein the distilled neural network is also configured to process an input to generate a respective score for each of the plurality of classes, comprising:
processing each training input of the set of multiple training inputs using the cumbersome neural network in accordance with the trained values of the cumbersome model parameters to generate a cumbersome target soft output for the training input;
wherein for each training input of the set of multiple training inputs, the cumbersome target soft output for the training input generated by the cumbersome neural network comprises a respective cumbersome target soft score for each of the plurality of classes generated by a last layer of the cumbersome neural network, wherein each cumbersome target soft score satisfies:
q
i
c
=
exp
⁡
(
z
i
c
T
c
)
∑
j
⁢
exp
⁡
(
z
j
c
T
c
)
wherein qic is the cumbersome target soft score for a class i, zic is a weighted combination of outputs of a previous layer of the cumbersome neural network received by the last layer of the cumbersome neural network, j ranges from 1 to a total number of classes in the plurality of classes, and Tc is a respective temperature constant which is greater than 1;
and
training the distilled neural network to, for each training input of the set of multiple training inputs, process the training input to generate a distilled soft output that matches the cumbersome target soft output generated by the cumbersome neural network by processing the same training input;
wherein for each training input of the set of multiple training inputs, the distilled soft output for the training input generated by the distilled neural network comprises a respective distilled soft score for each of the plurality of classes generated by a last layer of the distilled neural network, wherein each distilled soft score satisfies:
q
i
d
=
exp
⁡
(
z
i
d
T
d
)
∑
j
⁢
exp
⁡
(
z
j
d
T
d
)
wherein qid is the distilled soft score for a class i, zid is a weighted combination of outputs of a previous layer of the distilled neural network received by the last layer of the distilled neural network, j ranges from 1 to a total number of classes in the plurality of classes, and Td is a respective temperature constant which is greater than 1;
and
wherein during the training of the distilled neural network, the temperature constant Td corresponding to the distilled neural network is set to a higher value than used to generate distilled scores for the classes after the distilled neural network has been trained.

62 more elements

Training distilled machine learning models

Life of this patent

Training distilled machine learning models

Life of this patent

Abstract

Claims20