Systems and Methods of Distributed Optimization

A device for performing voice recognition, the device comprising:

• a microphone;
• a speaker;
• one or more processors;
• and
• one or more memory devices, the one or more memory devices storing:
• a machine-learned voice recognition model;
• and
• computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations, the operations comprising:
• determining a local model update associated with an objective function based at least in part on a number of local data examples stored by the device, the local model update being determined by performing a number of iterations of a gradient descent training technique on a local version of the machine-learned voice-recognition model with respect to the number of local data examples stored by the device;
• providing the local model update to a central computing system for use in determination of a global version of the machine-learned voice-recognition model based on aggregation of the local model update with one or more additional local model updates received from one or more additional devices, the aggregation of the local model update with the one or more additional local model updates comprising application of a weighting term to each local model update, the weighting term for each local model update being proportional to the number of local data examples stored on the corresponding device;
• receiving the global version of the machine-learned voice-recognition model from the central computing system;
• and
• employing the global version of the machine-learned voice-recognition model to produce voice recognition predictions based on voice data captured by the microphone.

A device for performing voice recognition, the device comprising:

• a microphone;
• a speaker;
• one or more processors;
• and
• one or more memory devices, the one or more memory devices storing computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations, the operations comprising:
• accessing a global version of a machine-learned voice-recognition model, wherein the global version of the machine-learned model has been generated based on aggregation of a plurality of local model updates respectively generated by a plurality of additional devices, the aggregation of the plurality of local model updates comprising application of a weighting term to each local model update, the weighting term for each local model update being proportional to a number of local data examples stored on the additional device that generated the local model update;
• capturing voice data with the microphone;
• and
• employing the global version of the machine-learned voice-recognition model to produce voice recognition predictions based on the voice data captured by the microphone.

A computer-implemented method of performing voice recognition, the method comprising:

• receiving, by a device that comprises a microphone and a speaker, a global version of a machine-learned voice-recognition model obtained from a central computing system, wherein the global version of the machine-learned model has been generated based on aggregation of a plurality of local model updates respectively generated by a plurality of additional devices, the aggregation of the plurality of local model updates comprising application of a weighting term to each local model update, the weighting term for each local model update being proportional to a number of local data examples stored on the additional device that generated the local model update;
• and
• capturing, by the device using the microphone, voice data;
• and
• employing, by the device, the received global version of the machine-learned voice recognition model to produce voice recognition predictions based on the voice data captured by the microphone.

A computer-implemented method of performing voice recognition, the method comprising:

• determining, by a first device that comprises a microphone, a speaker, and a machine-learned voice recognition model, a local model update to the machine-learned voice recognition model based at least in part on a gradient of a loss function and a number of locally stored data examples;
• providing, by the first device, the local model update to a central computing system for use in determination of a global update to the machine-learned voice recognition model by the central computing system via aggregation of the local model update with one or more additional local model updates received from one or more additional devices, the aggregation of the local model update with the one or more additional local model updates comprising application of a weighting term to each local model update, the weighting term for each local model update being proportional to the number of locally stored data examples stored on the corresponding device;
• receiving, by a second device that comprises a microphone and a speaker, the machine-learned voice recognition model from the central computing system after application of the global update to the machine-learned voice recognition model by the central computing system;
• and
• employing, by the second device, the received machine-learned voice recognition model to produce voice recognition predictions based on voice data captured by the microphone of the second device.