Skip-hop collective compute data transfer

A method for performing a collective compute operation on tensor data in a logical ring of processing ranks formed by a set of rank groups each containing m number of processing ranks including a primary rank and one or more secondary ranks, m being an integer greater than 1, the method comprising:

• at each rank group:
• receiving, at the primary rank of the rank group via an intranode interconnect of the rank group, an incoming data slice from a previous primary rank at m hops away on the logical ring;
• performing data transfer between the primary rank and each secondary rank of the rank group;
• and
• transferring an outgoing data slice from the primary rank of the rank group to a next primary rank at m hops away on the logical ring.

A non-transitory computer readable medium having stored therein instructions that, when executed by one or more processors, cause the one or more processors to perform operations for a collective computation on tensor data in a logical ring of processing ranks formed by a set of rank groups each containing m number of processing ranks including a primary rank and one or more secondary ranks, m being an integer greater than 1, the operations including:

• at each rank group:
• receiving, at the primary rank of the rank group via an intranode interconnect of the rank group, an incoming data slice from a previous primary rank at m hops away on the logical ring;
• performing data transfer between the primary rank and each secondary rank of the rank group;
• and
• transferring an outgoing data slice from the primary rank of the rank group to a next primary rank at m hops away on the logical ring.