# US 12,112,198 B2 · Asynchronous distributed data flow for machine learning workloads > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,198 B2 - Document ID: US12112198B2 - Status: Granted - Kind: B2 - Application number: 18/082,415 - Filed: 2022-12-15 - Published: 2023-04-20 - Granted: 2024-10-08 - Earliest claimed: 2021-05-07 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Jeffrey Adgate Dean; Sudip Roy; Michael Acheson Isard; Aakanksha Chowdhery; Brennan Saeta; Chandramohan Amyangot Thekkath; Daniel William Hurt; Hyeontaek Lim; Laurent El Shafey; Parker Edward Schuh; Paul Ronald Barham; Ruoming Pang; Ryan Sepassi; Sanjay Ghemawat; Yonghui Wu - Web page: https://patentgrep.com/patent/US12112198B2 ## CPC - G06F 9/4881: {Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues} - G06F 9/5016: {the resource being the memory} - G06F 9/5072: {Grid computing} - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/063: using electronic means - G06N 3/08: Learning methods - G06N 3/098: Distributed learning, e.g. federated learning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Methods, systems, and apparatus, including computer programs encoded on computer storage media, for distributing machine learning workloads, e.g., computations for training a neural network or computing an inference using a neural network, across multiple hardware accelerators. One of the systems comprises a plurality of accelerator islands, each hardware accelerator island comprising a respective plurality of hardware devices that include a plurality of hardware accelerators and a corresponding host for each of the plurality of hardware accelerators; and a respective scheduler for each of the accelerator islands that is configured to schedule workloads across the plurality of accelerators and corresponding hosts in the accelerator island, wherein the system is configured to: receive data representing a machine learning workload; and assign a respective portion of the machine learning workload to each of the plurality of accelerator islands for scheduling by the respective scheduler for the accelerator island. ## Claims ### Claim 1 A system comprising: a plurality of accelerator islands, each accelerator island comprising a respective plurality of hardware devices that include a plurality of hardware accelerators and a corresponding host for each of the plurality of hardware accelerators, wherein the hardware accelerators within each accelerator island are interconnected with one another over an interconnect network, and are connected to the hardware accelerators within another accelerator island over a data center network through their corresponding hosts; and a respective scheduler for each of the plurality of accelerator islands that is configured to schedule workloads across the plurality of accelerators and corresponding hosts in the accelerator island, wherein the system is configured to: receive data representing a machine learning workload; and assign a respective portion of the machine learning workload to each of the plurality of accelerator islands for scheduling by the respective scheduler for the accelerator island, wherein the respective scheduler is configured to, when the respective portion of the machine learning workload assigned to the accelerator island is a regular computation, schedule the respective portion of the machine learning workload using parallel asynchronous dispatch. ### Claim 2 (depends on 1) The system of claim 1, wherein the data representing the machine learning workload is data representing a sharded dataflow program comprising a plurality of shards. ### Claim 3 (depends on 2) The system of claim 2, wherein assigning the respective portion of the machine learning workload to each of the plurality of accelerator islands comprises assigning one or more shards of the sharded dataflow program to each of the plurality of accelerator islands. ### Claim 4 (depends on 1) The system of claim 1, wherein scheduling the respective portion of the computation using parallel asynchronous dispatch comprises: generating a schedule that assigns, to each of a set of the hardware accelerators in the accelerator island, a respective set of one or more operations that takes as input and output of one or more respective other operations that are performed by another one of the hardware accelerators in the accelerator island; determining, for each of the set of hardware accelerators, a respective size of the output of the one or more respective other operations; and transmitting, in parallel and to the corresponding host for each of the set of hardware accelerators, respective future data specifying the respective size of the output of the one or more respective other operations. ### Claim 5 (depends on 4) The system of claim 4, wherein the respective future data causes the corresponding host to (i) allocate memory on the hardware accelerator for storing the output of the one or more respective other operations and (ii) transmit data to a corresponding host of the accelerator assigned to the one or more respective other operations that identifies the allocated memory. ### Claim 6 (depends on 5) The system of claim 5, wherein the corresponding host of the accelerator assigned to the one or more respective other operations is configured to cause the accelerator assigned to the one or more respective other operations to transmit the output of the respective other operations to the allocated memory over the interconnect network. ### Claim 7 (depends on 1) The system of claim 1, wherein the interconnect network comprises an Inter-Core Interconnect (ICI) network. ### Claim 8 (depends on 1) The system of claim 1, wherein the data center network comprises an Ethernet network. ### Claim 9 One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the operations comprising: receiving data representing a machine learning workload by a plurality of accelerator islands, wherein each accelerator island comprises a respective plurality of hardware devices that include a plurality of hardware accelerators and a corresponding host for each of the plurality of hardware accelerators, wherein the hardware accelerators within each accelerator island are interconnected with one another over an interconnect network, and are connected to the hardware accelerators within another accelerator island over a data center network through their corresponding hosts, and wherein each accelerator island has a respective scheduler that is configured to schedule workloads across the plurality of accelerators and corresponding hosts in the accelerator island; and assigning a respective portion of the machine learning workload to each of the plurality of accelerator islands for scheduling by the respective scheduler for the accelerator island, wherein the respective scheduler is configured to, when the respective portion of the machine learning workload assigned to the accelerator island is a regular computation, schedule the respective portion of the computation using parallel asynchronous dispatch. ### Claim 10 (depends on 9) The non-transitory computer-readable storage media of claim 9, wherein the interconnect network comprises an Inter-Core Interconnect (ICI) network, and wherein the data center network comprises an Ethernet network. ### Claim 11 A method comprising: receiving data representing a machine learning workload by a plurality of accelerator islands, wherein each accelerator island comprises a respective plurality of hardware devices that include a plurality of hardware accelerators and a corresponding host for each of the plurality of hardware accelerators, wherein the hardware accelerators within each accelerator island are interconnected with one another over an interconnect network, and are connected to the hardware accelerators within another accelerator island over a data center network through their corresponding hosts, and wherein each accelerator island has a respective scheduler that is configured to schedule workloads across the plurality of accelerators and corresponding hosts in the accelerator island; and assigning a respective portion of the machine learning workload to each of the plurality of accelerator islands for scheduling by the respective scheduler for the accelerator island, wherein the respective scheduler is configured to, when the respective portion of the machine learning workload assigned to the accelerator island is a regular computation, schedule the respective portion of the computation using parallel asynchronous dispatch. ### Claim 12 (depends on 11) The method of claim 11, wherein the data representing the machine learning workload is data representing a sharded dataflow program comprising a plurality of shards. ### Claim 13 (depends on 12) The method of claim 12, wherein assigning the respective portion of the machine learning workload to each of the plurality of accelerator islands comprises assigning one or more shards of the sharded dataflow program to each of the plurality of accelerator islands. ### Claim 14 (depends on 11) The method of claim 11, wherein scheduling the respective portion of the computation using parallel asynchronous dispatch comprises: generating a schedule that assigns, to each of a set of the hardware accelerators in the accelerator island, a respective set of one or more operations that takes as input and output of one or more respective other operations that are performed by another one of the hardware accelerators in the accelerator island; determining, for each of the set of hardware accelerators, a respective size of the output of the one or more respective other operations; and transmitting, in parallel and to the corresponding host for each of the set of hardware accelerators, respective future data specifying the respective size of the output of the one or more respective other operations. ### Claim 15 (depends on 14) The method of claim 14, wherein the respective future data causes the corresponding host to (i) allocate memory on the hardware accelerator for storing the output of the one or more respective other operations and (ii) transmit data to a corresponding host of the accelerator assigned to the one or more respective other operations that identifies the allocated memory. ### Claim 16 (depends on 15) The method of claim 15, wherein the corresponding host of the accelerator assigned to the one or more respective other operations is configured to cause the accelerator assigned to the one or more respective other operations to transmit the output of the respective other operations to the allocated memory. ### Claim 17 (depends on 11) The method of claim 11, wherein the interconnect network comprises an Inter-Core Interconnect (ICI) network. ### Claim 18 (depends on 11) The method of claim 11, wherein the data center network comprises an Ethernet network.