US 2026/0223734 A1Published application

Artificial intelligence integrated circuit system configured with stacked three-dimensional memory devices

Assigneed-MATRIX CORPORATION

InventorsAayush ANKIT, Sudeep BHOJA

Life of this patent

Estimating the term…

Loading patent…

Abstract

A stacked AI engine system using three-dimensional (3D) memory devices. The system includes one or more memory integrated circuit (IC) devices coupled to a logic IC device in a stacked configuration. The memory ICs each include a plurality of first memory devices configured on a first semiconductor substrate and the logic IC includes multiple chiplets, each having multiple gangs, and each gang having multiple slices configured on a second semiconductor substrate. Each slice includes a first network-on-chip (NoC) device coupled to one of the first memory devices, a second NoC device coupled to a second memory device, multiple compute engine devices coupled to the NoC devices, an output buffer (OB) device coupled to compute engines, and a single instruction, multiple data (SIMD) device coupled to the OB device. These IC devices can be configured overlying an organic substrate member and coupled to each other via interconnections in underfill layers.

Claims24

Dependent claims

A stacked artificial intelligence (AI) engine system comprising:

one or more memory integrated circuit (IC) devices, each of the memory IC devices comprising
a first semiconductor substrate;
a plurality of first memory devices configured on the first semiconductor substrate, each of the memory devices having a plurality of bank groups, and each of the bank groups having a plurality of banks;
and
a logic integrated circuit (IC) device comprising
a second semiconductor substrate member;
a plurality of chiplets, each of the chiplets having a plurality of gangs, each of the gangs having a plurality of slices configured on the second semiconductor substrate member, and each of the slices comprising:
a first network-on-chip (NoC) device coupled to at least one of the plurality of first memory devices of one of the memory IC devices;
a second NoC device coupled to a second memory device;
a plurality of compute engine devices, each of the compute engines devices comprising an input buffer (IB) device coupled to the first NoC device, a weight buffer (WB) device coupled to the second NoC device, and a first compute device coupled to the IB device and the WB device, wherein the first compute device is configured to perform one or more matrix operations;
an output buffer (OB) device coupled to the first compute device;
a second compute device coupled to the OB device, wherein the second compute device is configured to perform one or more vector operations;
wherein the OB device is configured to store intermediate outputs of the matrix operations and the vector operations;
and
a first connector region having a first plurality of interconnections, the first connector region coupling the logic IC device to at least one of the memory IC devices in a stacked configuration.

12 more elements

A stacked artificial intelligence (AI) engine system comprising:

one or more dynamic random access memory (DRAM) integrated circuit (IC) devices, each of the DRAM IC devices comprising
a first semiconductor substrate;
a plurality of DRAM devices configured on the first semiconductor substrate, each of the DRAM devices having a plurality of bank groups, and each of the bank groups being configured as a channel and having a plurality of banks;
and
a logic IC device comprising
a second semiconductor substrate member;
a plurality of chiplets, each of the chiplets having a plurality of gangs, each of the gangs having a plurality of slices configured on the second semiconductor substrate member, and each of the slices comprising:
a weight network-on-chip (NoC) device coupled to at least one of the plurality of DRAM devices of one of the DRAM IC devices;
an input NoC device coupled to a static random access memory (SRAM) device;
a plurality of compute engine devices, each of the compute engines devices comprising an input buffer (IB) device coupled to the input NoC device, a weight buffer (WB) device coupled to the weight NoC device, and a first digital in-memory compute (DIMC) device coupled to the IB device and the WB device, wherein the DIMC device is configured to perform one or more matrix operations;
an output buffer (OB) device coupled to the DIMC device;
a single instruction, multiple data (SIMD) device coupled to the OB device, wherein the SIMD device is configured to perform one or more vector operations;
wherein the OB device is configured to store intermediate outputs of the matrix operations and the vector operations;
and
a first connector region having a first plurality of interconnections, the first connector region coupling the logic IC device to at least one of the memory IC devices in a stacked configuration.

12 more elements

Dependencies are inferred from the claim text.