Accelerated processing device and method of sharing data for machine learning

A method of data sharing comprising;

• receiving, at a first arithmetic logic unit among a plurality of arithmetic logic units, an instruction to perform a matrix multiplication calculation, wherein each of the plurality of arithmetic logic units are that communicatively coupled to at least a plurality of compute units and the plurality of arithmetic logic units each include a memory;
• when data, used to perform the matrix multiplication calculation, is available in the memory of the first arithmetic logic unit, accessing the data from the memory to perform the matrix multiplication calculation;
• and
• when the data is not available in the memory of the first arithmetic logic unit, but is available in the memory of a second arithmetic logic unit among the plurality of the arithmetic logic units, directly accessing the data from the memory of the second arithmetic logic unit via one or more interconnects connecting the plurality of arithmetic logic units, wherein the one or more interconnects enable direct data access by any of the arithmetic logic units to the memory of each of the plurality of arithmetic logic units.

A processing device comprising:

• a plurality of groups of compute units configured to process data;
• a plurality of arithmetic logic units that are each communicatively coupled to at least one of the plurality of group of the compute units, each arithmetic logic unit including a memory configured to store a portion of the data, wherein the plurality of arithmetic logic units are configured to perform calculations using the data stored among the plurality of arithmetic logic units;
• and
• a plurality of interconnect networks, each interconnect network configured to connect the arithmetic logic units of one of the groups of arithmetic logic units and to enable direct data access to the memory of each of the arithmetic logic units by any of the arithmetic logic units of the one group of arithmetic logic units.