Efficient and more advanced implementation of ring-allreduce algorithm for distributed parallel deep learning

A system for syncing data of a computing task across a plurality of groups of computing nodes, each group comprising a set of computing nodes A-D, a set of intra-group interconnects that communicatively couple computing node A with computing nodes B and C and computing node D with computing nodes B and C, and a set of inter-group interconnects that communicatively couple a computing node A of a first group of the plurality of groups with a computing node A of a second group neighboring the first group, and a computing node A of a third group neighboring the first group a computing node B of the first group with a computing node B of the second group and a computing node B of the third group, a computing node C of the first group with a computing node C of the second group and a computing node C of the third group, and a computing node D of the first group with a computing node D of the second group and a computing node D of the third group, wherein the second group and the third group are aligned with the first group in different dimensions, the system comprising:

• a memory storing a set of instructions;
• and
• one or more processors configured to execute the set of instructions to cause the system to:
• sync data across a first dimension of computing nodes of the first group and the second group using a first set of ring connections, wherein the first set of ring connections are formed using inter-group and intra-group interconnects that communicatively couple the computing nodes of the first group and the second group along the first dimension, and the syncing data across the first dimension comprises transferring, in a unit time, sub-data from a computing node along the first dimension to another computing node in a unit time via a connection on the first set of ring connections;
• sync data across a second dimension of computing nodes of the first group and the third group using a second set of ring connections, wherein the second set of ring connections are formed using inter-group and intra-group interconnects that communicatively couple the computing nodes of the first group and the third group along the second dimension, and syncing data across the second dimension comprises transferring, in a unit time, sub-data from a computing node along the second dimension to another computing node in a unit time via a connection on the second set of ring connections;
• and
• broadcast synced data across the first dimension or the second dimension of computing nodes using the first set of ring connections or the second set of ring connections.

A non-transitory computer readable medium that stores a set of instructions that is executable by one or more processors of an apparatus to cause the apparatus to initiate a method for syncing data of a computing task across a plurality of groups of computing nodes, each group comprising a set of computing nodes A-D, a set of intra-group interconnects that communicatively couple computing node A with computing nodes B and C and computing node D with computing nodes B and C, and a set of inter-group interconnects that communicatively couple a computing node A of a first group of the plurality of groups with a computing node A of a second group neighboring the first group and a computing node A of a third group neighboring the first group, a computing node B of the first group with a computing node B of the second group and a computing node B of the third group, a computing node C of the first group with a computing node C of the second group and a computing node C of the third group, and a computing node D of the first group with a computing node D of the second group and a computing node D of the third group, wherein the second group and the third group are aligned with the first group in different dimensions, the method comprising:

• syncing data across a first dimension of computing nodes of the first group and the second group using a first set of ring connections, wherein the first set of ring connections are formed using inter-group and intra-group interconnects that communicatively couple the computing nodes of the first group and the second group along the first dimension, and the syncing data across the first dimension comprises transferring, in a unit time, sub-data from a computing node along the first dimension to another computing node in a unit time via a connection on a first set of ring connections;
• syncing data across a second dimension of computing nodes of the first group and the third group using a second set of ring connections, wherein the second set of ring connections are formed using inter-group and intra-group interconnects that communicatively couple the computing nodes of the first group and the third group along the second dimension, and syncing data across the second dimension comprises transferring, in a unit time, sub-data from a computing node along the second dimension to another computing node in a unit time via a connection on the second set of ring connections;
• and
• broadcasting synced data across the first dimension or the second dimension of computing nodes using the first set of ring connections or the second set of ring connections.