# US 2024/0004646 A1 · Systems and methods for data placement for in-memory-compute

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0004646 A1
- Document ID: US20240004646A1
- Status: Published application
- Kind: A1
- Application number: 18/368,515
- Filed: 2023-09-14
- Published: 2024-01-04
- Earliest claimed: 2020-02-12
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Krishna T. Malladi; Wenqin Huangfu
- Web page: https://patentgrep.com/patent/US20240004646A1

## CPC

- G06F 9/3001: {Arithmetic instructions}
- G06F 7/5318: {with column wise addition of partial products, e.g. using Wallace tree, Dadda counters (G06F7/5324 takes precedence)}
- G06F 7/5443: {Sum of products (for applications thereof, see the relevant places, e.g. G06F17/10, H03H17/00)}
- G06F 7/57: Arithmetic logic units [ALU], i.e. arrangements or devices for performing two or more of the operations covered by groups G06F7/483 – G06F7/556 or for performing logical operations {(G06F7/49, G06F7/491 take precedence)}
- G06F 9/30036: {Instructions to perform operations on packed data, e.g. vector, tile or matrix operations}
- G06F 9/3004: {to perform operations on memory}
- G06F 9/30098: {Register arrangements}
- G06F 9/3016: {Decoding the operand specifier, e.g. specifier format}
- G06F 15/7821: {Tightly coupled to memory, e.g. computational memory, smart memory, processor in memory}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

According to one embodiment, a memory module includes: a memory die including a dynamic random access memory (DRAM) banks, each including: an array of DRAM cells arranged in pages; a row buffer to store values of one of the pages; an input/output (IO) module; and an in-memory compute (IMC) module including: an arithmetic logic unit (ALU) to receive operands from the row buffer or the IO module and to compute an output based on the operands and one of a plurality of ALU operations; and a result register to store the output of the ALU; and a controller to: receive, from a host processor, operands and an instruction; determine, based on the instruction, a data layout; supply the operands to the DRAM banks in accordance with the data layout; and control an IMC module to perform one of the ALU operations on the operands in accordance with the instruction.

## Claims

### Claim 1

A memory module comprising:
a dynamic random access memory (DRAM) device comprising:
an array of DRAM rows; and
an in-memory compute (IMC) module; and
a memory controller configured to:
receive, from a host processor, an input data and an instruction; and
supply the input data to the DRAM device in a data arrangement selected based on the instruction,
wherein the IMC module is configured to perform an operation on the input data based on the instruction.

### Claim 2 (depends on 1)

The memory module of claim 1, wherein the data arrangement specifies:
a first input data is read from the array of DRAM rows and the input data comprises a second input data, the second input data being supplied from the host processor to the IMC module.

### Claim 3 (depends on 2)

The memory module of claim 2, wherein the IMC module comprises an input buffer, and
wherein the memory controller is further configured to:
store a first portion of the first input data in the input buffer; and
perform the operation on the first input data stored in the input buffer and second portions of the second input data.

### Claim 4 (depends on 1)

The memory module of claim 1, wherein the input data is divided into first portions and second portions, a portion comprising values, and
wherein the data arrangement controls the memory controller to store one or more of the first portions and one or more of the second portions in a row of the array of DRAM rows.

### Claim 5 (depends on 1)

The memory module of claim 1, wherein the IMC module comprises an input buffer, and
wherein the memory controller is further configured to:
store a first portion of first portions of the input data in the input buffer, the first portions being stored in a first row of the array of DRAM rows; and
perform the operation on the first portion stored in the input buffer and second portions of the input data stored in the first row of the array of DRAM rows.

### Claim 6 (depends on 1)

The memory module of claim 1, wherein the IMC module comprises an accumulator configured to store an accumulated value, the accumulator being configured to:
receive an output of the operation; and
update the accumulator with the sum of the accumulated value and the output.

### Claim 7 (depends on 1)

The memory module of claim 1, wherein a first portion of the input data has a first number of values and a second portion of the input data has a second number of values,
wherein the IMC module comprises an output buffer configured to store an outer product of the first portion and the second portion.

### Claim 8 (depends on 1)

The memory module of claim 1, wherein a first portion of the input data has a first number of values and a second portion of the input data has a second number of values,
wherein the IMC module comprises an output buffer configured to store a tensor product of the first portion and the second portion.

### Claim 9 (depends on 1)

The memory module of claim 1, wherein the input data comprises first portions and second portions, a portion comprising values, and
wherein the data arrangement controls the memory controller to store a subset of the first portions in a first row of the array of DRAM rows and a subset of the second portions in a second row of the array of DRAM rows.

### Claim 10 (depends on 1)

The memory module of claim 1, wherein the IMC module further comprises an input buffer, and
wherein the memory controller is further configured to:
store a first portion of input data from a first row of the array of DRAM rows in the input buffer; and
perform the operation on the first portion stored in the input buffer and second portions of the input data stored in a second row of the array of DRAM rows.

### Claim 11 (depends on 1)

The memory module of claim 1, wherein the IMC module comprises a hardware buffer configured to buffer an output of the operation.

### Claim 12 (depends on 1)

The memory module of claim 1, wherein the IMC module comprises an accumulator configured to store an accumulated value, the accumulator being configured to:
receive an output of the operation, the operation comprising an inner product operation; and
update the accumulator with the sum of the accumulated value and the output.

### Claim 13 (depends on 1)

The memory module of claim 1, wherein the memory module is a high bandwidth memory (HBM) module comprising a stack of memory dies connected by through silicon vias.

### Claim 14

A method for performing computations in-memory comprising:
receiving, by a memory controller of a memory module, an input data and an instruction;
determining, by the memory controller, based on the instruction, a data arrangement;
supplying the input data to a dynamic random access memory (DRAM) device in the data arrangement, the DRAM device comprising:
an array of DRAM rows; and
an in-memory compute (IMC) module,
wherein the IMC module is configured to perform an operation on the input data based on the instruction,
wherein the data arrangement specifies placement of the input data in the array of DRAM rows and the IMC module.

### Claim 15 (depends on 14)

The method of claim 14, wherein the data arrangement specifies:
a first input data is read from the DRAM rows and the input data comprises a second input data supplied from a host processor to the IMC module.

### Claim 16 (depends on 14)

The method of claim 14, wherein the IMC module further comprises an input buffer, and
wherein the memory controller is further configured to:
store a first portion of a first input data in the input buffer; and
perform the operation on the first input data stored in the input buffer and second portions of a second operand stored in the array of DRAM rows.

### Claim 17 (depends on 14)

The method of claim 14, wherein the input data is divided into first portions and second portions, a portion comprising values, and
wherein the data arrangement controls the memory controller to store one or more of the first portions and one or more of the second portions in a first row of the DRAM rows.

### Claim 18 (depends on 14)

The method of claim 14, wherein the IMC module comprises an input buffer, and
wherein the memory controller is further configured to:
store a first portion of one or more of the first portions of the input data in the input buffer, the first portions of the input data being stored in a first row of the array of DRAM rows; and
perform the operation on the first portion stored in the input buffer and second tiles stored in the first row of the array of DRAM rows.

### Claim 19 (depends on 14)

The method of claim 14, wherein the input data comprises first portions and second portions, a portion comprising values, and
wherein the data arrangement controls the memory controller to store a subset of the first portion in a first row of the array of DRAM rows and a subset of the second portions in a second row of the array of DRAM rows.

### Claim 20 (depends on 14)

The method of claim 14, wherein the IMC module further comprises an input buffer, and
wherein the memory controller is further configured to:
store a first portion of the input data from a first row of the array of DRAM rows in the input buffer; and
perform the operation on the first portion stored in the input buffer and second portions of the input data stored in a second row of the array of DRAM rows.
