Systems and Methods of Distributed Optimization

A computer-implemented method of updating a global model based on unevenly distributed data, the method comprising:

• providing, by one or more computing devices, a current global model to a plurality of user devices;
• receiving, by the one or more computing devices, a plurality of local updates to the current global model from the plurality of user devices, each local update being determined by the respective user device through performance of a respective number of iterations of a gradient descent training technique on the current global model with respect to a respective number of data examples stored on the respective user device, and wherein the one or more data examples stored on the plurality of user devices are distributed on an uneven basis, such that no user device includes a representative sample of an overall distribution of data examples;
• and
• aggregating, by the one or more computing devices, the received local updates to determine an updated global model, wherein aggregating, by the one or more computing devices, the received local updates to determine the updated global model comprises applying, by the one or more computing devices, a respective weighting term to each local update, and wherein the respective weighting term for each local update is proportional to the respective number of data examples stored on the user device from which such local update was received.

A computer-implemented method of updating a local machine learning model based on unevenly distributed data, the method comprising:

• determining, by a user device, a local model update based at least in part on a gradient vector of a loss function and a number of locally stored data examples, wherein the distribution of the number of locally stored data examples is not representative of an overall distribution of data examples used to train a global machine learning model;
• providing, by the user device, the local model update to a central computing device for use in determination of an update to the global machine learning model, the update to the global machine learning model being determined based on aggregation of the local model update with one or more additional local model updates received from one or more additional user devices, the aggregation of the local model update with the one or more additional local model updates comprising application of a weighting term to the local model update, the weighting term for the local model update being proportional to the number of locally stored data examples stored on the user device;
• after determination of the update to the global machine learning model, receiving, by the user device, the global machine learning model from the central computing device;
• and
• employing, by the user device, the global machine learning model to produce predictions.

A computing system, comprising:

• one or more processors;
• and
• one or more memory devices, the one or more memory devices storing computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations, the operations comprising:
• determining a local model update associated with an objective function based at least in part on a number of local data examples stored by the computing system, the local model update being determined by performing a number of iterations of a gradient descent training technique on a local version of a model with respect to the number of local data examples stored by the computing system;
• providing the local model update to a central computing device for use in the determination of a global version of the model based on aggregation of the local model update with one or more additional local model updates received from one or more additional computing systems, the aggregation of the local model update with the one or more additional local model updates comprising application of a weighting term to the local model update, the weighting term for the local model update being proportional to the number of local data examples stored on the computing system;
• receiving the global version of the model from the central computing device;
• and
• employing the global version of the model to produce predictions;
• wherein a distribution of the number of local data examples is not representative of an overall distribution of data examples stored on the one or more additional computing systems.