Asynchronous optimization for sequence training of neural networks

One or more non-transitory computer-readable media storing software that includes instructions, which, when executed by one or more computers, cause the one or more computers to perform operations comprising:

• obtaining multiple copies of a neural network of a speech model;
• asynchronously obtaining parameter values for the multiple copies of the neural network such that different copies of the neural network have different sets of parameter values;
• after obtaining the parameter values such that different copies of the neural network have different sets of parameter values, training the multiple copies of the neural network in parallel using different subsets of a set of training data, wherein training each copy of the neural network adjusts the parameter values for the copy of the neural network to generate adjusted parameter values;
• and
• updating the neural network of the speech model based on the adjusted parameter values generated for each of the multiple copies of the neural network.

A system comprising:

• one or more processors and one or more computer storage media storing instructions that are operable, when executed by the one or more processors, to cause the one or more processors to perform operations comprising:
• obtaining multiple copies of a neural network of a speech model;
• asynchronously obtaining parameter values for the multiple copies of the neural network such that different copies of the neural network have different sets of parameter values;
• after obtaining the parameter values such that different copies of the neural network have different sets of parameter values, training the multiple copies of the neural network in parallel using different subsets of a set of training data, wherein training each copy of the neural network adjusts the parameter values for the copy of the neural network to generate adjusted parameter values;
• and
• updating the neural network of the speech model based on the adjusted parameter values generated for each of the multiple copies of the neural network.