Control of data transfer between processing nodes

A method implemented in a data processing system having a plurality of processing nodes, a first processing node of the plurality of processing nodes comprising a memory configured to store an array of data items, wherein the first processing node belongs to a first set of processing nodes and to a second set of processing nodes, wherein the first processing node is the only processing node that is common to the first set of processing nodes and the second set of processing nodes, wherein the first processing nodes comprises:

• a processing unit;
• and
• a gateway device connected to the processing unit, the gateway device configured to interface the first processing node with a host storage, and wherein the method comprises, at the first processing node:
• taking part in a reduce-scatter collective using the array of data items to obtain a first reduced subset of the data items, wherein the taking part in the reduce-scatter collective is performed by the processing unit of the first processing node, and wherein the reduce-scatter collective is further performed between a second processing node and a third processing node of the second set of processing nodes;
• exchanging the first reduced subset of the data items by participating in an all-reduce collective to obtain a second reduced subset of the data items, wherein the exchanging the first reduced subset is performed by the gateway device of the first processing node, and wherein the all-reduce collective is performed among the first set of processing nodes;
• and
• taking part in an all-gather collective using the second reduced subset of the data items to obtain a reduced array of data items, wherein the taking part in the all-gather collective is performed by the processing unit of the first processing node, and wherein the all-gather collective is performed with the second processing node and the third processing node of the second set of processing nodes,
• wherein each of the plurality of first processing nodes executes compute instructions during a compute phase and, following an internal barrier synchronisation, enters an external barrier synchronisation including at least one exchange phase, wherein each of the plurality of first processing nodes:
• takes part in the reduce-scatter collective during the at least one exchange phase;
• performs the exchange of the respective reduced subset of the array of data items by participating in the all-reduce collective during the at least one exchange phase;
• and
• takes part in the one or more all-gather collectives during the at least one exchange phase.

A non-transitory computer readable storage medium storing a computer program comprising computer readable instructions, which when executed by a first processing node within a processing system having a plurality of processing nodes, causes the first processing node to:

• take part in a reduce-scatter collective using an array of data items to obtain a first reduced subset of the data items, wherein the taking part in the reduce-scatter collective is performed by a processing unit of the first processing node, and wherein the first processing node is associated with a first set of processing nodes and with a second set of processing nodes, wherein the first processing node is the only processing node that is common to the first set of processing nodes and the second set of processing nodes, and wherein the reduce-scatter collective is further performed between a second processing node and a third processing node of the second set of processing nodes;
• exchange the first reduced subset of the data items by participating in an all-reduce collective to obtain a second reduced subset of the data items, and wherein the all-reduce collective is performed among the first set of processing nodes;
• and
• take part in an all-gather collective using the second reduced subset of the data items to obtain a reduced array of data items, wherein the taking part in the all-gather collective is performed by the processing unit of the first processing node, and wherein the all-gather collective is performed with the second processing node and the third processing node of the second set of processing nodes,
• wherein the computer readable instructions, when executed by the at least one processor of the first processing node within a processing system having a plurality of processing nodes, further cause the first processing node to:
• execute compute instructions during a compute phase and, following an internal barrier synchronisation, enter an external barrier synchronisation including at least one exchange phase,
• take part in the reduce-scatter collective during the at least one exchange phase;
• perform the exchange of the respective reduced subset of the array of data items by participating in the all-reduce collective during the at least one exchange phase;
• and
• take part in the one or more all-gather collectives during the at least one exchange phase.