Serving Large Language Models with 3D-DRAM Chiplets
AssigneeGoogle Inc.
InventorsXiaoyu Ma, David Patterson
Life of this patent
Estimating the term…
Abstract
Disclosed systems and methods herein provide for high bandwidth processing using a plurality of compute-memory chiplets. A computing package may be configured with a plurality of the compute-memory chiplets in order to perform processing operations in connection with a large language model. The compute-memory chiplets may be configured to operate using small, low-power computing dies that can efficiently operate for workloads with low arithmetic intensity.
Claims20
A computing package comprising:
- a package substrate;
- and
- one or more computing clusters located on the package substrate;
- wherein each of the one or more computing clusters includes a plurality of compute-memory stacks in communication with an input-output die;
- wherein each compute-memory stack includes a plurality of memory dies stacked with a low-power compute die;
- and
- wherein the input-output die is configured to transmit data for the plurality of compute-memory stacks via one or more peripheral component interconnects.
A method of computing comprising:
- receiving processing commands at one or more computing clusters located on a package substrate;
- performing computing operations based on the processing commands using a plurality of compute-memory stacks in communication with an input-output die, wherein each compute-memory stack includes a plurality of memory dies stacked with a low-power compute die;
- and
- transmitting data from the input-output die via one or more peripheral component interconnects.
A large model processing unit comprising one or more computing packages connected via peripheral component interconnects, each computing package comprising:
- a package substrate;
- and
- one or more computing clusters located on the package substrate;
- wherein each of the one or more computing clusters includes a plurality of compute-memory stacks in communication with an input-output die;
- wherein each compute-memory stack includes a plurality of memory dies stacked with a low-power compute die;
- and
- wherein the input-output die is configured to transmit data for the plurality of compute-memory stacks via the one or more peripheral component interconnects.
Dependencies are inferred from the claim text.