Privacy-sensitive training of machine learning models

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for privacy-sensitive training of a machine learning model, the machine learning model comprising a set of model parameters, values of which are updated over a plurality of training iterations to optimize an objective function, the operations comprising, at each of the training iterations:

• obtaining a respective batch of training data items;
• using the training data items in the batch to determine a gradient of the objective function for the current training iteration;
• obtaining a respective plurality of noise values;
• and
• updating the values of the model parameters using the gradient of the objective function for the current training iteration and the plurality of noise values, and wherein updating the values of the model parameters using the gradient of the objective function for the current training iteration comprises:
• updating a set of supplementary values of the model parameters using the gradient of the objective function for the current training iteration in accordance with a first learning rate;
• updating the values of the model parameters using the gradient of the objective function for the current training iteration in accordance with a second learning rate that differs from the first learning rate;
• and
• further updating the values of the model parameters by combining the values of the model parameters with the supplementary values of the model parameters.

A system comprising:

• one or more computers;
• and
• one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for privacy-sensitive training of a machine learning model, the machine learning model comprising a set of model parameters, values of which are updated over a plurality of training iterations to optimize an objective function, the operations comprising, at each of the training iterations:
• obtaining a respective batch of training data items;
• using the training data items in the batch to determine a gradient of the objective function for the current training iteration;
• obtaining a respective plurality of noise values;
• and
• updating the values of the model parameters using the gradient of the objective function for the current training iteration and the plurality of noise values, and
• wherein updating the values of the model parameters using the gradient of the objective function for the current training iteration comprises:
• updating a set of supplementary values of the model parameters using the gradient of the objective function for the current training iteration in accordance with a first learning rate;
• updating the values of the model parameters using the gradient of the objective function for the current training iteration in accordance with a second learning rate that differs from the first learning rate;
• and
• further updating the values of the model parameters by combining the values of the model parameters with the supplementary values of the model parameters.