US 12,417,047 B2 · Heterogeneous ML accelerator cluster with flexible system resource balance

Abstract

Aspects of the disclosure are directed to a heterogeneous machine learning accelerator system with compute and memory nodes connected by high speed chip-to-chip interconnects. While existing remote/disaggregated memory may require memory expansion via remote processing units, aspects of the disclosure add memory nodes into machine learning accelerator clusters via the chip-to-chip interconnects without needing assistance from remote processing units to achieve higher performance, simpler software stack, and/or lower cost. The memory nodes may support prefetch and intelligent compression to enable the use of low cost memory without performance degradation.

Claims20

Dependent claims

A machine learning accelerator system, comprising:

one or more accelerator clusters comprising a plurality of non-uniformly distributed memory nodes and a plurality of compute nodes connected by a high speed chip-to-chip interconnect that facilitates transfer of machine learning models or data, the non-uniformly distributed memory nodes having varying capacities or latencies;
and
one or more processors configured to:
keep a portion of the machine learning models or data uncompressed in faster memory nodes of the plurality of memory nodes and a portion of the machine learning models or data compressed in slower memory nodes of the plurality of memory nodes;
and
based on a latency threshold for the non-uniformly distributed memory nodes, prefetch the machine learning models or data from the slower memory nodes of the plurality of non-uniformly distributed memory nodes to on-chip memory in the compute nodes via the high speed interconnect.

3 more elements

A method for a machine learning accelerator system, the machine learning accelerator system including one or more accelerator clusters including a plurality of non-uniformly distributed memory nodes and a plurality of compute nodes connected by a high speed chip-to-chip interconnect that facilitates transfer of machine learning models or data, the non-uniformly distributed memory nodes having varying capacities or latencies, the method comprising:

determining, by one or more processors, memory nodes of the plurality of memory nodes constitute slower memory nodes based on a latency threshold for the non-uniformly distributed memory nodes;
keeping, by the one or more processors, a portion of the machine learning models or data uncompressed in faster memory nodes of the plurality of memory nodes and a portion of the machine learning models or data compressed in the slower memory nodes of the plurality of memory nodes;
and
prefetching, by the one or more processors, the machine learning models or data from the slower memory nodes to on-chip memory in the compute nodes via the high speed interconnect.

1 more element

A non-transitory computer readable medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for a machine learning accelerator system, the machine learning accelerator system including one or more accelerator clusters including a plurality of non-uniformly distributed memory nodes and a plurality of compute nodes connected by a high speed chip-to-chip interconnect that facilitates transfer of machine learning models or data, the non-uniformly distributed memory nodes having varying capacities or latencies, the operations comprising:

determining memory nodes of the plurality of memory nodes constitute slower memory nodes based on a latency threshold for the non-uniformly distributed memory nodes;
keeping, by the one or more processors, a portion of the machine learning models or data uncompressed in faster memory nodes of the plurality of memory nodes and a portion of the machine learning models or data compressed in the slower memory nodes of the plurality of memory nodes;
and
prefetching the machine learning models or data from the slower memory nodes to on-chip memory in the compute nodes via the high speed interconnect.

1 more element

Dependencies are inferred from the claim text.

Heterogeneous ML accelerator cluster with flexible system resource balance

Life of this patent

Heterogeneous ML accelerator cluster with flexible system resource balance

Life of this patent

Abstract

Claims20