Graphic processor unit topology-aware all-reduce operation

A computer program product comprising a computer readable storage medium having program instructions for executing a topology-aware all-reduce method embodied therewith, the program instructions executable by an environment including a plurality of systems and a switch in communication with the plurality of systems, each system of the plurality of systems including a plurality of computing modules comprising a plurality of graphic processing units (GPUs), the program instructions executable to cause:

• locally partitioning and scattering data slices among the plurality of computing modules of each system to produce local summation results;
• internally copying the local summation results from the plurality of computing modules to corresponding host memories of the plurality of systems;
• facilitating by the switch, a cross system all-reduce operation among the plurality of systems to cause an exchange of the local summation results across the host memories and a determination of final summation partitions from the local summation results;
• internally copying the final summation partitions from the host memories to the corresponding plurality of computing modules of each system;
• forwarding the final summation partitions to all GPUs to cause a determination of final summation results therefrom;
• and
• executing an optimized pipelining scheme to create multiple iterations of the topology-aware all-reduce operation,
• wherein the topology-aware all-reduce method is asynchronous across the environment.

An environment including a plurality of systems and a switch in communication with the plurality of systems, each system of the plurality of systems including a plurality of computing modules comprising a plurality of graphic processing units (GPUs), the environment executing a topology-aware all-reduce operation comprising:

• locally partitioning and scattering data slices among the plurality of computing modules of each system to produce local summation results;
• internally copying the local summation results from the plurality of computing modules to corresponding host memories of the plurality of systems;
• facilitating by the switch, a cross system all-reduce operation among the plurality of systems to cause an exchange of the local summation results across the host memories and a determination of final summation partitions from the local summation results;
• internally copying the final summation partitions from the host memories to the corresponding plurality of computing modules of each system;
• forwarding the final summation partitions to all GPUs to cause a determination of final summation results therefrom;
• and
• executing an optimized pipelining scheme to create multiple iterations of the topology-aware all-reduce operation,
• wherein the topology-aware all-reduce method is asynchronous across the environment.