Control of processing node operations

A method implemented in a data processing system having a data storage and a plurality of processing nodes configured to perform operations to implement a process for performing an iteration of a neural network over a plurality of time periods, wherein the method comprises:

• a first processing node of the plurality of processing nodes, during a first time period of the plurality of time periods, loading weights associated with a second layer of the neural network and required during a subsequent time period of the plurality of time periods for performing second calculations to determine activations of the second layer of the neural network from the data storage;
• a second processing node of the plurality of processing nodes, during the first time period, performing first calculations to determine activations of a first layer of the neural network using weights associated with the first layer loaded from the data storage during an earlier one of the plurality of time periods;
• the first processing node, receiving the activations of the first layer from the second processing node;
• during the subsequent time period, performing the second calculations using the activations of the first layer and using the weights associated with the second layer loaded during the first time period;
• wherein the plurality of processing nodes are configured to, as part of the process, use the activations of the second layer of the neural network to determine activations of one or more further layers of the neural network, including determining output values of the neural network,
• wherein the plurality of processing nodes are configured to use the process to perform training of the neural network by:
• determining the output values of the neural network by performing the process for performing the iteration of the neural network;
• comparing the output values of the neural network to labels of the neural network to calculate loss;
• and
• performing a backward propagation through the neural network of the loss to produce a revised set of weights of the neural network;
• receiving, by the first processing node, the activations of the first layer from the second processing node during an exchange phase of the first processing node and the second processing node, wherein each of the plurality of time periods corresponds to a compute phase of the second processing node;
• and
• executing, by the second processing node, a compiled code sequence comprising a synchronisation instruction indicating a barrier between the compute phase and the exchange phase.

A non-transitory computer readable medium storing a computer program comprising sets of computer readable instructions, wherein when each of the sets of computer readable instructions are executed causes a data processing system to perform operations to implement a process for performing an iteration of a neural network over a plurality of time periods, the operations comprising:

• during a first time period of the plurality of time periods, load weights associated with a second layer of the neural network and required during a subsequent time period of the plurality of time periods for performing second calculations to determine activations of the second layer of the neural from a data storage, wherein loading the load weights associated with the second layer is performed by a first processing node of the data processing system;
• during the first time period, perform first calculations to determine activations of a first layer of the neural network using weights associated with the first layer loaded from the data storage during an earlier one of the plurality of time periods, wherein the first calculations are performed by a second processing node of the data processing system;
• receive the weights associated with the second layer of the neural network at the second processing node from the first processing node;
• during the subsequent time period, perform the second calculations using the weights associated with the second layer of the neural network,
• wherein the plurality of processing nodes are configured to, as part of the process, use the activations of the second layer of the neural network to determine activations of one or more further layers of the neural network, including determining output values of the neural network, and
• wherein the plurality of processing nodes are configured to use the process to perform training of the neural network by:
• determining the output values of the neural network by performing the process for performing the iteration of the neural network;
• comparing the output values of the neural network to labels of the neural network to calculate loss;
• and
• performing a backward propagation through the neural network of the loss to produce a revised set of weights of the neural network;
• and
• receive, by the first processing node, the activations of the first layer from the second processing node during an exchange phase of the first processing node and the second processing node, wherein each of the plurality of time periods corresponds to a compute phase of the second processing node;
• and
• execute, by the second processing node, a compiled code sequence comprising a synchronisation instruction indicating a barrier between the compute phase and the exchange phase.