Training a model using parameter server shards

A method for training a model having parameters by determining a respective parameter value for each of the parameters of the model, the method comprising:

• receiving, at a parameter server shard configured to maintain values of a disjoint partition of the parameters of the model, a succession of respective requests for parameter values from each of a plurality of replicas of the model, wherein each of the plurality of replicas is an identical instance of the model with possibly different parameter values for the parameters of the model;
• in response to each request, downloading, by the parameter server shard, a current value of each requested parameter to the replica from which the request was received;
• receiving, by the parameter server shard and from each of the plurality of replicas, a succession of uploads, each upload including respective delta values for each of the parameters in the partition maintained by the shard;
• and
• updating, by the parameter server shard, values of the parameters in the partition maintained by the parameter server shard repeatedly based on the uploads of delta values to generate current parameter values.

A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by one or more computers cause the one or more computers to perform operations for training a model having parameters by determining a respective parameter value for each of the parameters of the model, the operations comprising:

• receiving, at a parameter server shard configured to maintain values of a disjoint partition of the parameters of the model, a succession of respective requests for parameter values from each of a plurality of replicas of the model, wherein each of the plurality of replicas is an identical instance of the model with possibly different parameter values for the parameters of the model;
• in response to each request, downloading a current value of each requested parameter to the replica from which the request was received;
• receiving, from each of the plurality of replicas, a succession of uploads, each upload including respective delta values for each of the parameters in the partition maintained by the shard;
• and
• updating values of the parameters in the partition maintained by the parameter server shard repeatedly based on the uploads of delta values to generate current parameter values.