System Having Multiple Processing Unit Sets For Training Neural Networks

A data processing system for training a neural network, the data processing system comprising:

• a first set of one or more processing units, a second set of one or more processing units, at least one data storage, and at least one interconnect between the first set of one or more processing units, the second set of processing units and the at least one data storage,
• wherein the at least one data storage is configured to provide over the at least one interconnect, training data to the first set of one or more processing units and the second set of one more processing units,
• wherein each of the first and second set of processing units is configured to, for each of at least some of a plurality of training iterations for training the neural network:
• perform a series of operations on at least part of the training data from the at least one data storage to derive output values for the neural network;
• exchange over the at least one interconnect, with the other of the first and second set of processing units, the output values calculated by the respective one of the first and second set of processing units;
• evaluate a loss function for the respective training iteration, said loss function including a metric measuring the dissimilarity between the output values calculated by the first and second set of processing units, wherein the metric is weighted in the evaluation of the loss function in accordance with a parameter;
• update model parameters of the neural network using the respective evaluated loss function;
• and
• update the parameter for use in subsequent ones of the training iterations.