US 12,566,962 B2Granted

Dithered quantization of parameters during training with a machine learning tool

AssigneeMicrosoft Technology Licensing, LLC

InventorsThomas M. Annau, Haishan Zhu, Daniel Lo, Eric S. Chung

Life of this patent

Estimating the term…

Loading patent…

Abstract

A machine learning tool uses dithered quantization of parameters during training of a machine learning model such as a neural network. The machine learning tool receives training data and initializes certain parameters of the machine learning model (e.g., weights for connections between nodes of a neural network, biases for nodes). The machine learning tool trains the parameters in one or more iterations based on the training data. In particular, in a given iteration, the machine learning tool applies the machine learning model to at least some of the training data and, based at least in part on the results, determines parameter updates to the parameters. The machine learning tool updates the parameters using the parameter updates and a dithered quantizer function, which can add random values before a rounding or truncation operation.

Claims20

Dependent claims

A method, implemented in a computer system comprising at least one hardware processor and at least one memory coupled to the at least one hardware processor, the method comprising:

receiving training data;
initializing parameters of a machine learning model;
training at least some of the parameters in multiple iterations based on the training data, including, in each of the multiple iterations during the training:
applying the machine learning model to at least some of the training data;
based at least in part on results of the applying the machine learning model, determining a measure of loss as a difference between an output of the machine learning model for the at least some of the training data and an expected output of the machine learning model for the at least some of the training data;
based at least in part on the measure of loss for the at least some of the training data, determining a set of parameter updates to the at least some of the parameters;
combining predetermined respective dither values from a random value generator function with respective combinations of respective parameter updates of the set of parameter updates and respective starting parameter values of respective parameters of the at least some of the parameters used during the applying the machine learning model to at least some of the training data, wherein at least a portion of the respective predetermined dither values differ from one another, to provide respective quantizer function operands;
and
after the combining the predetermined respective dither values, quantizing the respective quantizer function operands using a quantizer function to provide one or more updated parameter values for at least a portion of the at least some of the parameters, wherein the one or more updated parameter values are used in at least one subsequent iteration of the multiple iterations.

6 more elements

A computer system comprising:

a buffer, in memory of the computer system, configured to receive training data;
and
a machine learning tool, implemented with one or more processors of the computer system, configured to perform operations comprising:
initializing parameters of a machine learning model;
training at least some of the parameters in multiple iterations based on the training data, including, in each of the multiple iterations during the training:
applying the machine learning model to at least some of the training data;
based at least in part on results of the applying the machine learning model, determining a measure of loss as a difference between an output of the machine learning model for the at least some of the training data and an expected output of the machine learning model for the at least some of the training data;
based at least in part on the measure of loss for the at least some of the training data, determining a set of parameter updates to the at least some of the parameters;
combining predetermined respective dither values from a random value generator function with respective combinations of respective parameter updates of the set of parameter updates and respective starting parameter values of respective parameters of the at least some of the parameters used during the applying the machine learning model to at least some of the training data, wherein at least a portion of the respective predetermined dither values differ from one another, to provide respective quantizer function operands;
after the combining the predetermined respective dither values, quantizing the respective quantizer function operands using a quantizer function to provide one or more updated parameter values for at least a portion of the at least some of the parameters, wherein the one or more updated parameter values are used in at least one subsequent iteration of training of the machine learning model;
and
a buffer, in memory of the computer system, configured to store the parameters.

9 more elements

One or more computer-readable media having stored thereon computer-executable instructions for causing one or more processing units, when programmed thereby, to perform operations comprising:

receiving training data;
initializing parameters of a machine learning model;
training at least some of the parameters in multiple iterations based on the training data, including, in each of the multiple iterations during the training:
applying the machine learning model to at least some of the training data;
based at least in part on results of the applying the machine learning model, determining a measure of loss as a difference between an output of the machine learning model for the at least some of the training data and an expected output of the machine learning model for the at least some of the training data;
based at least in part on the measure of loss for the at least some of the training data, determining a set of parameter updates to the at least some of the parameters;
combining predetermined respective dither values from a random value generator function with respective combinations of respective parameter updates of the set of parameter updates and respective starting parameter values of respective parameters of the at least some of the parameters used during the applying the machine learning model to at least some of the training data, wherein at least a portion of the respective predetermined dither values differ from one another, to provide respective quantizer function operands;
and
after the combining the predetermined respective dither values, quantizing the respective quantizer function operands using a quantizer function to provide one or more updated parameter values for at least a portion of the at least some of the parameters, wherein the one or more updated parameter values are used in at least one subsequent iteration of training of the machine learning model.

6 more elements

Dependencies are inferred from the claim text.