Hardware implemented point to point communication primitives for machine learning

A graphics processing unit comprising:

• a compute block including one or more processing clusters, the one or more processing clusters to perform compute operations associated with a machine learning framework workflow, the compute operations to generate gradient data associated with trainable parameters of a neural network;
• a memory device to store gradient data during distributed training of the neural network;
• and
• a fabric interface coupled with the memory device, the fabric interface to transmit gradient data stored in the memory device, wherein the memory device is a physical memory device shared with the compute block.

A method of computing and distributing data for distributed training of a neural network, the method comprising:

• storing a first set of instructions in a first memory, the first set of instructions including a machine learning framework;
• enabling, via a fabric interface, transmission and receipt of data associated with a set of trainable machine learning parameters;
• executing the first set of instructions via a set of general-purpose processor cores, wherein executing the first set of instructions includes providing a training workflow for computation of gradients for the set of trainable machine learning parameters and communicating with a second set of instructions, the second set of instructions facilitating transmission and receipt of the gradients via the fabric interface;
• and
• performing, via a graphics processor, computing operations associated with the training workflow, wherein performing the computing operations includes generating the gradients for the set of trainable machine learning parameters.