# US 2018/0322386 A1 · Fine-grain compute communication execution for deep learning frameworks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0322386 A1
- Document ID: US20180322386A1
- Status: Published application
- Kind: A1
- Application number: 15/869,502
- Filed: 2018-01-12
- Published: 2018-11-08
- Earliest claimed: 2017-05-05
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: SRINIVAS SRIDHARAN; DHEEVATSA MUDIGERE
- Web page: https://patentgrep.com/patent/US20180322386A1

## CPC

- G06N 3/098: Distributed learning, e.g. federated learning
- G06F 9/54: Interprogram communication
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/063: using electronic means
- G06N 3/09: Supervised learning
- G06T 15/005: {General purpose rendering architectures}
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06T 15/04: Texture mapping
- G06T 15/80: Shading
- G06T 17/10: Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

One embodiment provides for a system to configure distributed training of a neural network. The system includes memory to store a library to facilitate transmission of data during distributed training of the neural network; a network interface to transmit and receive gradient data associated with the trainable parameters; a general-purpose processor to execute instructions provided by the library, the instructions to cause the general-purpose processor to configure the network interface to transmit and receive the gradient data associated with the trainable parameters during a workflow of a machine learning framework; and a graphics processor to perform compute operations associated with machine learning framework workflow to generate the gradient data associated with the trainable parameters, wherein, based on the machine learning framework workflow, the library is to interleave the compute operations on the graphics processor with transmission and receipt of gradient data via the network interface.

## Claims

### Claim 1

A system to configure distributed training of a neural network, the system comprising:
memory to store a library to facilitate transmission of data during distributed training of the neural network, the data associated with trainable parameters of the neural network;
a network interface to transmit and receive gradient data associated with the trainable parameters;
a general-purpose processor to execute instructions provided by the library, the instructions to cause the general-purpose processor to configure the network interface to transmit and receive the gradient data associated with the trainable parameters during a workflow of a machine learning framework; and
a graphics processor to perform compute operations associated with machine learning framework workflow to generate the gradient data associated with the trainable parameters, wherein, based on the machine learning framework workflow, the library is to interleave the compute operations on the graphics processor with transmission and receipt of gradient data via the network interface.

### Claim 2 (depends on 1)

The system as in claim 1, wherein a compute operation is configured to overlap with a communication operation to send or receive gradient data via the network interface.

### Claim 3 (depends on 2)

The system as in claim 2, the machine learning framework workflow to cause the graphics processor to perform a compute operation associated with a first portion of a first layer of the neural network.

### Claim 4 (depends on 3)

The system as in claim 3, wherein in response to a notification of completion of the compute operation associated with the first portion of the first layer of the neural network, the library is to cause the network interface to transmit a result of the compute operation.

### Claim 5 (depends on 4)

The system as in claim 4, the network interface to transmit the result according to a communication pattern for messages to be transmitted between worker nodes during distributed training of the neural network.

### Claim 6 (depends on 5)

The system as in claim 5, wherein the communication pattern is a gather, scatter, allgather, alltoall, reduce, reduce scatter, or all-reduce.

### Claim 7 (depends on 1)

The system as in claim 1, wherein the network interface is a fabric interface to enable a connection to a communication fabric, the communication fabric to interconnect worker nodes of a distributed training network.

### Claim 8 (depends on 7)

The system as in claim 7, wherein the fabric interface is a peripheral component interconnect express interface.

### Claim 9 (depends on 7)

The system as in claim 7, wherein the fabric interface is an NVLink interface.

### Claim 10 (depends on 7)

The system as in claim 7, wherein the graphics processor includes at least a portion of the fabric interface.

### Claim 11

A graphics processing unit comprising:
a compute block including one or more processing clusters, the one or more processing clusters to perform compute operations associated with a machine learning framework workflow, the compute operations to generate gradient data associated with trainable parameters of a neural network;
a memory device to store gradient data during distributed training of the neural network; and
a fabric interface coupled with the memory device, the fabric interface to transmit and receive gradient data associated with the trainable parameters.

### Claim 12 (depends on 11)

The graphics processing unit as in claim 11, the fabric interface including a cache to store gradient data before transmission or after receipt of the gradient data.

### Claim 13 (depends on 12)

The graphics processing unit as in claim 12, wherein the cache memory is associated with the memory device.

### Claim 14 (depends on 11)

The graphics processing unit as in claim 11, wherein the fabric interface has an address space mapped to at least a portion of the memory device.

### Claim 15 (depends on 14)

The graphics processing unit as in claim 14, wherein the compute block and the fabric interface have a unified address space.

### Claim 16

A method of performing distributed training of a neural network, the method comprising:
storing a library in memory, the library to facilitate transmission of data during distributed training of the neural network, the data associated with trainable parameters of the neural network;
transmitting and receiving gradient data associated with the trainable parameters via a network interface;
executing, via a general-purpose processor, instructions provided by the library, the instructions to cause the general-purpose processor to configure the network interface to transmit and receive the gradient data associated with the trainable parameters during a workflow of a machine learning framework; and
performing, via a graphics processor, compute operations associated with a machine learning framework workflow to generate the gradient data associated with the trainable parameters, the compute operations interleaved with transmission and receipt of gradient data via the network interface.

### Claim 17 (depends on 16)

The method as in claim 16, additionally comprising configuring a compute operation to overlap with a communication operation to send or receive gradient data via the network interface.

### Claim 18 (depends on 17)

The method as in claim 17, additionally comprising performing, via the graphics processor, a compute operation associated with a first portion of a first layer of the neural network.

### Claim 19 (depends on 18)

The method as in claim 18, wherein in response to a notification of completion of the compute operation associated with the first portion of the first layer of the neural network, causing, via the library, the network interface to transmit a result of the compute operation.

### Claim 20 (depends on 19)

The method as in claim 19, additionally comprising transmitting the result according to a communication pattern for messages to be transmitted between worker nodes during distributed training of the neural network.