Fine-grain compute communication execution for deep learning frameworks

A graphics processing unit comprising:

• a compute block including one or more processing clusters, the one or more processing clusters to perform compute operations associated with a machine learning framework workflow, the compute operations to generate gradient data associated with trainable parameters of a neural network;
• a memory device to store gradient data during distributed training of the neural network;
• and
• a fabric interface coupled with the memory device, the fabric interface to transmit and receive gradient data associated with the trainable parameters.

A method of performing distributed training of a neural network, the method comprising:

• storing a library in memory, the library to facilitate transmission of data during distributed training of the neural network, the data associated with trainable parameters of the neural network;
• transmitting and receiving gradient data associated with the trainable parameters via a network interface;
• executing, via a general-purpose processor, instructions provided by the library, the instructions to cause the general-purpose processor to configure the network interface to transmit and receive the gradient data associated with the trainable parameters during a workflow of a machine learning framework;
• and
• performing, via a graphics processor, compute operations associated with a machine learning framework workflow to generate the gradient data associated with the trainable parameters, the compute operations interleaved with transmission and receipt of gradient data via the network interface.