System Having Multiple Processing Unit Sets For Training Neural Networks

A method for training a neural network, the method implemented in a data processing system comprising:

• a first set of processing units, a second set of processing units, data storage, and an interconnect between the first set of processing units, the second set of processing units and the data storage, wherein the first set of processing units is configured to run a first model of the neural network and the second set of processing units is configured to run a second model of the neural network, wherein the method comprises:
• calculating first output values for the first model of the neural network from training data;
• calculating second output values for the second model of the neural network from the training data;
• exchanging over the interconnect, data indicating a state of the first model and data indicating a state of the second model between the first set of processing units and the second set of processing units;
• determining, based on the data indicating the state of the second model, third output values calculated for the second model;
• evaluating a loss function for a first training iteration, the loss function including a measure of dissimilarity between the second output values and the third output values, wherein the measure of dissimilarity is weighted in the evaluation of the loss function in accordance with a first parameter;
• and
• updating model parameters of the neural network using the loss function.

A non-transitory computer readable medium comprising a computer program which when executed by a data processing system carries out a method for training a neural network, the method comprising:

• calculating first output values for a first model of the neural network from training data;
• calculating second output values for a second model of the neural network from the training data;
• exchanging data indicating a state of the first model and data indicating a state of the second model between a first set of processing units and a second set of processing units;
• determining, based on the data indicating the state of the second model, third output values calculated for the second model;
• evaluating a loss function for a first training iteration, the loss function including a measure of dissimilarity between the second output values and the third output values, wherein the measure of dissimilarity is weighted in the evaluation of the loss function in accordance with a first parameter;
• and
• updating model parameters of the neural network using the loss function.