US 2026/0056891 A1 · STACKED APPARATUS USING IN-MEMORY COMPUTE CHIPLET DEVICES FOR INFERENCE-TIME COMPUTE ACCELERATION

Abstract

A stacked apparatus using in-memory compute (IMC) chiplet devices for inference-time compute acceleration. The apparatus is configured to accelerate the workload computations for neural network models, such as those for Large Language Models (LLMs) and reasoning models. The apparatus achieves high throughput and low latency using a chiplet design, digital IMC (DIMC) based engines, efficient die-to-die (D2D) interconnects, block floating point (BFP) numerics, and large high bandwidth on-chip memories. With modular chiplets in stacked configurations with memory devices and efficient interconnects, the accelerator apparatus can be easily scaled to accelerate workloads for models of different sizes. The DIMC configuration within the chiplet slices also improves computational performance and reduces power consumption by integrating computational functions and memory fabric. And by dynamically switching between precision levels based on real-time analysis of a target workload, computational efficiency can be optimized while maintaining accuracy.

Claims20

Dependent claims

A stacked in-memory compute (IMC) accelerator apparatus comprising:

an interposer substrate;
a plurality of chiplets spatially configured overlying the interposer substrate, each of the chiplets comprising a plurality of tiles, and each of the tiles comprising:
a plurality of slices, and
a central processing unit (CPU) coupled to the plurality of slices;
a plurality of die-to-die (D2D) interconnects coupled to the CPUs in each of the tiles through the interposer substrate;
and
a memory interface coupled to the CPUs in each of the tiles;
and
a plurality of memory devices coupled to the plurality of chiplets via the memory interface such that each chiplet and one or more memory devices form a stacked configuration;
wherein each of the slices includes a plurality of compute cores, each compute core include a digital IMC (DIMC) device and a Single Input Multiple Data (SIMD) device coupled to an output buffer (OB) device;
wherein each of the slices includes a stash memory device coupled to the DIMC device in each of the compute cores;
wherein each of the slices includes a global memory (GM) device coupled to the plurality of compute cores;
and
wherein each of the slices includes a data reshape engine (DRE) device coupled to the plurality of compute cores.

11 more elements

A stacked in-memory compute (IMC) accelerator apparatus comprising:

an interposer substrate;
a plurality of chiplets spatially configured overlying the interposer substrate, each of the chiplets comprising a plurality of tiles, and each of the tiles comprising:
a tile crossbar device;
a plurality of slices coupled to the tile crossbar device, and
a tile central processing unit (CPU) coupled to the plurality of slices;
a plurality of die-to-die (D2D) interconnects coupled to the each of the tile CPUs through the interposer substrate;
a memory interface coupled to each of the tile CPUs;
and
a plurality of memory devices coupled to the plurality of chiplets via the memory interface such that each chiplet and one or more memory devices form a stacked configuration;
and
wherein each of the slices comprises
a slice crossbar device coupled to tile crossbar device;
a plurality of compute cores coupled to the slice crossbar device, wherein each compute core includes a digital IMC (DIMC) device and a Single Input Multiple Data (SIMD) device coupled to an output buffer (OB) device;
a global memory (GM) device coupled to slice crossbar device;
a stash memory device coupled to the slice crossbar device and the DIMC devices of the compute cores;
and
a data reshape engine (DRE) device coupled to the slice crossbar device.

14 more elements

A stacked in-memory compute (IMC) apparatus comprising:

a switch device coupled to a first stacked IMC accelerator apparatus and a second stacked IMC accelerator apparatus;
wherein each of the first and second stacked IMC accelerator apparatuses comprises a plurality of chiplets configured overlying an interposer substrate, each of the chiplets comprising a plurality of tiles, and each of the tiles comprising:
a tile crossbar device;
a plurality of slices coupled to the tile crossbar device, and
a tile central processing unit (CPU) coupled to the plurality of slices;
a plurality of die-to-die (D2D) interconnects coupled to the each of the tile CPUs through the interposer substrate;
a memory interface coupled to each of the tile CPUs;
and
a plurality of memory devices coupled to the plurality of chiplets via the memory interface such that each chiplet and one or more memory devices form a stacked configuration;
and
wherein each of the slices comprises
a slice crossbar device coupled to tile crossbar device;
a plurality of compute cores coupled to the slice crossbar device, wherein each compute core includes a digital IMC (DIMC) device and a Single Input Multiple Data (SIMD) device coupled to an output buffer (OB) device;
a global memory (GM) device coupled to slice crossbar device;
a stash memory device coupled to the slice crossbar device and the DIMC devices of the compute cores;
and
a data reshape engine (DRE) device coupled to the slice crossbar device;
and
a bridge connection interface coupled to at least one of the chiplets of the first stacked IMC accelerator apparatus and at least one of the chiplets of the second stacked IMC accelerator apparatus, the bridge connection interface being configured to provide one or more back-to-back links between the first and second stacked IMC accelerator apparatuses.

16 more elements

Dependencies are inferred from the claim text.

Stacked apparatus using in-memory compute chiplet devices for inference-time compute acceleration

Life of this patent

Stacked apparatus using in-memory compute chiplet devices for inference-time compute acceleration

Life of this patent

Abstract

Claims20