Differentially private stochastic gradient descent using optimized correlation matrices

A system comprising:

• one or more computers;
• and
• one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations, the operations comprising:
• training the neural network on a set of training examples to perform a machine learning task, the training comprising, at each of a plurality of training iterations:
• determining gradients of an objective function with respect to a set of neural network parameters of the neural network;
• generating a plurality of noise values for modifying the gradients of the objective function, comprising:
• stochastically sampling a plurality of initial noise values from a noise distribution that is parametrized by a noise multiplier;
• and
• generating the noise values by applying a correlation matrix to the plurality of initial noise values;
• generating noisy gradients by combining the plurality of noise values with the gradients of the objective function;
• and
• updating the set of neural network parameters of the neural network using the noisy gradients;
• wherein the correlation matrix used during the training of the neural network is determined by performing operations comprising:
• iteratively updating a current correlation matrix over a plurality of optimization iterations to optimize a utility metric, comprising, at each optimization iteration:
• determining a current noise multiplier that, if used in conjunction with the current correlation matrix during the training of the neural network, is predicted to cause the trained neural network to satisfy a data security criterion;
• determining a current utility metric for the current correlation matrix based at least in part on the current noise multiplier;
• determining gradients of the current utility metric with respect to the current correlation matrix;
• and
• updating the current correlation matrix using the gradients of the current utility metric.

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations, the operations comprising:

• training the neural network on a set of training examples to perform a machine learning task, the training comprising, at each of a plurality of training iterations:
• determining gradients of an objective function with respect to a set of neural network parameters of the neural network;
• generating a plurality of noise values for modifying the gradients of the objective function, comprising:
• stochastically sampling a plurality of initial noise values from a noise distribution that is parametrized by a noise multiplier;
• and
• generating the noise values by applying a correlation matrix to the plurality of initial noise values;
• generating noisy gradients by combining the plurality of noise values with the gradients of the objective function;
• and
• updating the set of neural network parameters of the neural network using the noisy gradients;
• wherein the correlation matrix used during the training of the neural network is determined by performing operations comprising:
• iteratively updating a current correlation matrix over a plurality of optimization iterations to optimize a utility metric, comprising, at each optimization iteration:
• determining a current noise multiplier that, if used in conjunction with the current correlation matrix during the training of the neural network, is predicted to cause the trained neural network to satisfy a data security criterion;
• determining a current utility metric for the current correlation matrix based at least in part on the current noise multiplier;
• determining gradients of the current utility metric with respect to the current correlation matrix;
• and
• updating the current correlation matrix using the gradients of the current utility metric.