Hyper-square implementation of tree allreduce algorithm for distributed parallel deep learning

A system for syncing data of a computing task across a plurality of groups of computing nodes, each group comprising a set of computing nodes A-D, a set of intra-group interconnects that communicatively couple computing node A with computing nodes B and C and computing node D with computing nodes B and C, and a set of inter-group interconnects that communicatively couple a computing node A of a first group with a computing node A of a second group neighboring the first group, a computing node B of the first group with a computing node B of the second group, a computing node C of the first group with a computing node C of the second group, and a computing node D of the first group with a computing node D of the second group, the system comprising:

• a memory storing a set of instructions;
• and
• one or more processors configured to execute the set of instructions to cause the system to:
• sync data at a computing node of the plurality of groups of computing nodes using inter-group interconnects and intra-group interconnects along four different directions relative to the node, wherein the four directions for syncing data comprises two horizontal-based directions and two vertical-based directions and two directions of the four different directions involve intra-group interconnects and two other directions of the four different directions involve inter-group interconnects;
• and
• broadcast synced data from the node to the plurality of groups of computing nodes using inter-group interconnects and intra-group interconnects along four different directions relative to the node, wherein the four directions for broadcasting data comprises two horizontal-based directions and two vertical-based directions and two directions of the four different directions involve intra-group interconnects and two other directions of the four different directions involve inter-group interconnects.

A non-transitory computer readable medium that stores a set of instructions that is executable by one or more processors of an apparatus to cause the apparatus to initiate a method for syncing data of a computing task across a plurality of groups of computing nodes, each group comprising a set of computing nodes A-D, a set of intra-group interconnects that communicatively couple computing node A with computing nodes B and C and computing node D with computing nodes B and C, and a set of inter-group interconnects that communicatively couple a computing node A of a first group with a computing node A of a second group neighboring the first group, a computing node B of the first group with a computing node B of the second group, a computing node C of the first group with a computing node C of the second group, and a computing node D of the first group with a computing node D of the second group, the method comprising:

• syncing data at a computing node of the plurality of groups of computing nodes using inter-group interconnects and intra-group interconnects along four different directions relative to the node, wherein the four directions for syncing data comprises two horizontal-based directions and two vertical-based directions and two directions of the four different directions involve intra-group interconnects and two other directions of the four different directions involve inter-group interconnects;
• and
• broadcasting synced data from the node to the plurality of groups of computing nodes using inter-group interconnects and intra-group interconnects along four different directions relative to the node, wherein the four directions for broadcasting data comprises two horizontal-based directions and two vertical-based directions and two directions of the four different directions involve intra-group interconnects and two other directions of the four different directions involve inter-group interconnects.