# US 2021/0240543 A1 · Efficient and more advanced implementation of ring-allreduce algorithm for distributed parallel deep learning > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0240543 A1 - Document ID: US20210240543A1 - Status: Published application - Kind: A1 - Application number: 16/777,711 - Filed: 2020-01-30 - Published: 2021-08-05 - Earliest claimed: 2020-01-30 - Assignees: Alibaba Group Holding Limited - Applicants: Alibaba Group Holding Limited - Inventors: Liang HAN; Jeff JIAO - Web page: https://patentgrep.com/patent/US20210240543A1 ## CPC - G06F 9/52: Program synchronisation; Mutual exclusion, e.g. by means of semaphores - G06F 9/4881: {Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues} - G06N 3/063: using electronic means - G06N 3/08: Learning methods - G06N 3/098: Distributed learning, e.g. federated learning - G06N 20/00: Machine learning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract The present disclosure provides a method for syncing data of a computing task across a plurality of groups of computing nodes, each group comprising a set of computing nodes A-D, a set of intra-group interconnects that communicatively couple computing node A with computing nodes B and C and computing node D with computing nodes B and C, and a set of inter-group interconnects that communicatively couple a computing node A of a first group of the plurality of groups with a computing node A of a second group neighboring the first group, a computing node B of the first group with a computing node B of the second group, a computing node C of the first group with the computing node C of the second group, and a computing node D of the first group with a computing node D of the second group, the method comprising: syncing across a first dimension of computing nodes using a first set of ring connections, wherein the first set of ring connections are formed using inter-group and intra-group interconnects that communicatively couple the computing nodes along the first dimension; and broadcasting synced data across a second dimension of computing nodes using a second ring connection. ## Claims ### Claim 1 A method for syncing data of a computing task across a plurality of groups of computing nodes, each group comprising a set of computing nodes A-D, a set of intra-group interconnects that communicatively couple computing node A with computing nodes B and C and computing node D with computing nodes B and C, and a set of inter-group interconnects that communicatively couple a computing node A of a first group of the plurality of groups with a computing node A of a second group neighboring the first group and a computing node A of a third group neighboring the first group, a computing node B of the first group with a computing node B of the second group and a computing node B of the third group, a computing node C of the first group with a computing node C of the second group and a computing node C of the third group, and a computing node D of the first group with a computing node D of the second group and a computing node D of the third group, wherein the second group and the third group are aligned with the first group in different dimensions, the method comprising: syncing data across a first dimension of computing nodes of the first group and the second group using a first set of ring connections, wherein the first set of ring connections are formed using inter-group and intra-group interconnects that communicatively couple the computing nodes of the first group and the second group along the first dimension, and the syncing data across the first dimension comprises transferring, in a unit time, sub-data from a computing node along the first dimension to another computing node in a unit time via a connection on the first set of ring connections; syncing data across a second dimension of computing nodes of the first group and the third group using a second set of ring connections, wherein the second set of ring connections are formed using inter-group and intra-group interconnects that communicatively couple the computing nodes of the first group and the third group along the second dimension, and syncing data across the second dimension comprises transferring, in a unit time, sub-data from a computing node along the second dimension to another computing node in a unit time via a connection on the second set of ring connections; and broadcasting synced data across the first dimension or the second dimension of computing nodes using the first set of ring connections or the second set of ring connections. ### Claim 2 (depends on 1) The method of claim 1, wherein the data to be synced comprises a plurality of sub-data, and each computing node comprises a different version of each sub-data. ### Claim 3 (depends on 2) The method of claim 2, wherein: syncing data across a first dimension of computing nodes using a first set of ring connections further comprises: in a clock cycle, receiving a version of sub-data by each computing node in a row, wherein the version of the sub-data is transferred from another computing node in the row via the connection on the first set of ring connections; and continuing data transferring until each computing node along the first dimension receives all versions of a sub-data from all computing nodes in the row. ### Claim 4 (depends on 3) The method of claim 3, wherein: syncing data across a second dimension of computing nodes using a second set of ring connections further comprises: in a clock cycle, receiving a version of sub-data by each computing node in a column, wherein the version of the sub-data is transferred from another computing node in the column via the connection on the second set of ring connections; and continuing data transferring until each computing node along the second dimension receives all versions of a sub-data from all computing nodes. ### Claim 5 (depends on 3) The method claim 3, wherein broadcasting synced data across the first dimension or the second dimension of computing nodes using the first set of ring connections or the second set of ring connections further comprises: in a clock cycle, receiving sub-data by each computing node across the first dimension or the second dimension, wherein the sub-data transferred from another computing node across the first dimension or the second dimension via the connection on the first set of ring connections or the connection on the second set of ring connections; and continuing data transferring until all computing nodes along the first dimension or the second dimension receive sub-data from all computing nodes. ### Claim 6 (depends on 1) The method of claim 1, wherein the set of intra-group interconnects and the set of inter-group interconnects comprise inter-chip interconnects. ### Claim 7 (depends on 1) The method of claim 1, wherein the computing nodes are artificial intelligence (“AI”) training processors, AI training chips, neural processing units (“NPU”), or graphic processing units (“GPU”). ### Claim 8 (depends on 7) The method of claim 7, wherein the computing task is an AI computing task involving an allreduce algorithm. ### Claim 9 A system for syncing data of a computing task across a plurality of groups of computing nodes, each group comprising a set of computing nodes A-D, a set of intra-group interconnects that communicatively couple computing node A with computing nodes B and C and computing node D with computing nodes B and C, and a set of inter-group interconnects that communicatively couple a computing node A of a first group of the plurality of groups with a computing node A of a second group neighboring the first group, and a computing node A of a third group neighboring the first group a computing node B of the first group with a computing node B of the second group and a computing node B of the third group, a computing node C of the first group with a computing node C of the second group and a computing node C of the third group, and a computing node D of the first group with a computing node D of the second group and a computing node D of the third group, wherein the second group and the third group are aligned with the first group in different dimensions, the system comprising: a memory storing a set of instructions; and one or more processors configured to execute the set of instructions to cause the system to: sync data across a first dimension of computing nodes of the first group and the second group using a first set of ring connections, wherein the first set of ring connections are formed using inter-group and intra-group interconnects that communicatively couple the computing nodes of the first group and the second group along the first dimension, and the syncing data across the first dimension comprises transferring, in a unit time, sub-data from a computing node along the first dimension to another computing node in a unit time via a connection on the first set of ring connections; sync data across a second dimension of computing nodes of the first group and the third group using a second set of ring connections, wherein the second set of ring connections are formed using inter-group and intra-group interconnects that communicatively couple the computing nodes of the first group and the third group along the second dimension, and syncing data across the second dimension comprises transferring, in a unit time, sub-data from a computing node along the second dimension to another computing node in a unit time via a connection on the second set of ring connections; and broadcast synced data across the first dimension or the second dimension of computing nodes using the first set of ring connections or the second set of ring connections. ### Claim 10 (depends on 9) The system of claim 9, wherein the data to be synced comprises a plurality of sub-data, and each computing node comprises a different version of each sub-data. ### Claim 11 (depends on 10) The system of claim 10, wherein the one or more processors are further configured to execute the set of instructions to cause the system to: in a clock cycle, receive a version of sub-data by each computing node in a row, wherein the version of the sub-data is transferred from another computing node in the row via the connection on the first set of ring connections; and continue data transferring until each computing node along the first dimension receives all versions of a sub-data from all computing nodes in the row. ### Claim 12 (depends on 11) The system of claim 11, wherein the one or more processors are further configured to execute the set of instructions to cause the system to: in a clock cycle, receive a version of sub-data by each computing node in a column, wherein the version of the sub-data is transferred from another computing node in the column via the connection on the second set of ring connections; and continue data transferring until each computing node along the second dimension receives all versions of a sub-data from all computing nodes. ### Claim 13 (depends on 10) The system of claim 10, wherein the one or more processors are further configured to execute the set of instructions to cause the system to: in a clock cycle, receive sub-data by each computing node across the first dimension or the second dimension, wherein the sub-data transferred from another computing node across the first dimension or the second dimension via the connection on the first set of ring connections or the connection on the second set of ring connections; and continue data transferring until all computing nodes along the first dimension or the second dimension receive sub-data from all computing nodes. ### Claim 14 (depends on 9) The system of claim 9, wherein the set of intra-group interconnects and the set of inter-group interconnects comprise inter-chip interconnects. ### Claim 15 (depends on 9) The system of claim 9, wherein the computing nodes are artificial intelligence (“AI”) training processors, AI training chips, neural processing units (“NPU”), or graphic processing units (“GPU”). ### Claim 16 (depends on 15) The system of claim 15, wherein the computing task is an AI computing task involving an allreduce algorithm. ### Claim 17 A non-transitory computer readable medium that stores a set of instructions that is executable by one or more processors of an apparatus to cause the apparatus to initiate a method for syncing data of a computing task across a plurality of groups of computing nodes, each group comprising a set of computing nodes A-D, a set of intra-group interconnects that communicatively couple computing node A with computing nodes B and C and computing node D with computing nodes B and C, and a set of inter-group interconnects that communicatively couple a computing node A of a first group of the plurality of groups with a computing node A of a second group neighboring the first group and a computing node A of a third group neighboring the first group, a computing node B of the first group with a computing node B of the second group and a computing node B of the third group, a computing node C of the first group with a computing node C of the second group and a computing node C of the third group, and a computing node D of the first group with a computing node D of the second group and a computing node D of the third group, wherein the second group and the third group are aligned with the first group in different dimensions, the method comprising: syncing data across a first dimension of computing nodes of the first group and the second group using a first set of ring connections, wherein the first set of ring connections are formed using inter-group and intra-group interconnects that communicatively couple the computing nodes of the first group and the second group along the first dimension, and the syncing data across the first dimension comprises transferring, in a unit time, sub-data from a computing node along the first dimension to another computing node in a unit time via a connection on a first set of ring connections; syncing data across a second dimension of computing nodes of the first group and the third group using a second set of ring connections, wherein the second set of ring connections are formed using inter-group and intra-group interconnects that communicatively couple the computing nodes of the first group and the third group along the second dimension, and syncing data across the second dimension comprises transferring, in a unit time, sub-data from a computing node along the second dimension to another computing node in a unit time via a connection on the second set of ring connections; and broadcasting synced data across the first dimension or the second dimension of computing nodes using the first set of ring connections or the second set of ring connections. ### Claim 18 (depends on 17) The non-transitory computer readable medium of claim 17, wherein the data to be synced comprises a plurality of sub-data, and each computing node comprises a different version of each sub-data. ### Claim 19 (depends on 18) The non-transitory computer readable medium of claim 18, wherein the method further comprises: in a clock cycle, receiving a version of sub-data by each computing node in a row, wherein the version of the sub-data is transferred from another computing node in the row via the connection on the first set of ring connections; and continuing data transferring until each computing node along the first dimension receives all versions of a sub-data from all computing nodes in the row. ### Claim 20 (depends on 19) The non-transitory computer readable medium of claim 19, wherein the method further comprises: in a clock cycle, receiving a version of sub-data by each computing node in a column, wherein the version of the sub-data is transferred from another computing node in the column via the connection on the second set of ring connections; and continuing data transferring until each computing node along the second dimension receives all versions of a sub-data from all computing nodes. ### Claim 21 (depends on 19) The non-transitory computer readable medium of claim 19, wherein the method further comprises: in a clock cycle, receiving sub-data by each computing node across the first dimension or the second dimension, wherein the sub-data transferred from another computing node across the first dimension or the second dimension via the connection on the first set of ring connections or the connection on the second set of ring connections; and continuing data transferring until all computing nodes along the first dimension or the second dimension receives sub-data from all computing nodes. ### Claim 22 (depends on 17) The non-transitory computer readable medium of claim 17, wherein the set of intra-group interconnects and the set of inter-group interconnects comprise inter-chip interconnects. ### Claim 23 (depends on 17) The non-transitory computer readable medium of claim 17, wherein the computing nodes are artificial intelligence (“AI”) training processors, AI training chips, neural processing units (“NPU”), or graphic processing units (“GPU”). ### Claim 24 (depends on 23) The non-transitory computer readable medium of claim 23, wherein the computing task is an AI computing task involving an allreduce algorithm.