# US 11,899,741 B2 · Memory device and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,899,741 B2
- Document ID: US11899741B2
- Status: Granted
- Kind: B2
- Application number: 16/856,380
- Filed: 2020-04-23
- Published: 2021-03-25
- Granted: 2024-02-13
- Earliest claimed: 2019-09-19
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Hyung-Dal Kwon; Seung Wook Lee
- Web page: https://patentgrep.com/patent/US11899741B2

## CPC

- G06F 17/15: Correlation function computation {including computation of convolution operations (arithmetic circuits for sum of products per se, e.g. multiply-accumulators G06F7/5443; digital filters, e.g. FIR, IIR, adaptive filters H03H17/00)}
- G06F 7/5443: {Sum of products (for applications thereof, see the relevant places, e.g. G06F17/10, H03H17/00)}
- G06F 13/28: using burst mode transfer, e.g. direct memory access {DMA}, cycle steal (G06F13/32 takes precedence)
- G06F 17/14: Fourier, Walsh or analogous domain transformations {, e.g. Laplace, Hilbert, Karhunen-Loeve, transforms (for correlation function computation G06F17/156; spectrum analysers G01R23/16)}
- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/063: using electronic means
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- H03H 17/0213: {Frequency domain filters using Fourier transforms}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H03H: IMPEDANCE NETWORKS, e.g. RESONANT CIRCUITS; RESONATORS (waveguides, resonators, lines or other devices of the waveguide type H01P)

## Abstract

A memory device includes a memory configured to store input data and filter data for a convolution operation, and a function processor configured to, in response to a read command of at least a portion of data from among the input data and the filter data, transform the at least a portion of the data based on a parameter of the convolution operation during a clock cycle corresponding to the read command and output a corresponding transformation result as transformed data.

## Claims

### Claim 1

A memory device comprising:
a memory configured to store input data and filter data for a convolution operation; and
a function processor, embedded inside the memory device, configured to:
in response to a read command of at least a portion of data from among the input data and the filter data, transform at least a first portion of the input data using a first transforming matrix and transform at least a second portion of the filter data using a second transforming matrix, respectively during a clock cycle corresponding to the read command, the first transforming matrix and the second transforming matrix being respectively based on a parameter of the convolution operation, and output a corresponding transformation result, as transformed data, using a third transforming matrix; and
reduce information loss associated with the outputting by adjusting relationship between the second transforming matrix and the third transforming matrix.

### Claim 2 (depends on 1)

The memory device of claim 1, wherein the function processor comprises an input data transform circuit configured to transform the at least first portion of the input data and output a corresponding transformation result as transformed input data, and
the input data transform circuit is structured based on the parameter of the convolution operation and a type of an algorithm applied to transform the at least first portion of the input data to reduce a computational quantity of the convolution operation.

### Claim 3 (depends on 2)

The memory device of claim 2, wherein an input transform matrix is determined based on both the parameter of the convolution operation and the type of the algorithm, and
the input data transform circuit is structured to correspond to the input transform matrix.

### Claim 4 (depends on 1)

The memory device of claim 1, wherein the function processor comprises a filter data transform circuit configured to transform the at least second portion of the filter data and output a corresponding transformation result as transformed filter data, and
the filter data transform circuit is structured based on the parameter of the convolution operation and a type of an algorithm applied to transform the at least second portion of the filter data to reduce a computational quantity of the convolution operation, and
wherein a filter transform matrix is determined, as the second transforming matrix, based on both the parameter of the convolution operation and the type of the algorithm, and the filter data transform circuit is structured to correspond to the filter transform matrix.

### Claim 5 (depends on 1)

The memory device of claim 1, wherein the filter data stored in the memory comprises filter data transformed based on the parameter of the convolution operation, and
when the filter data is transformed based on the parameter of the convolution operation, the function processor is configured to output the transformed filter data without further transforming the transformed filter data.

### Claim 6 (depends on 1)

The memory device of claim 1, wherein the function processor comprises a multiplier accumulator (MAC) configured to perform an operation between the transformed input data and the transformed filter data.

### Claim 7 (depends on 1)

The memory device of claim 1, wherein the function processor is configured to, in response to a write command of at least a portion of intermediate output data that is output through an operation between the transformed input data and the transformed filter data, transform the at least portion of the intermediate output data based on the parameter of the convolution operation during a clock cycle corresponding to the write command, and output a corresponding transformation result as transformed intermediate output data.

### Claim 8 (depends on 7)

The memory device of claim 7, wherein the intermediate output data comprises output data of an elementwise multiplication between the transformed input data and the transformed filter data.

### Claim 9 (depends on 7)

The memory device of claim 7, wherein the function processor comprises an output data transform circuit configured to transform the at least portion of the intermediate output data and output a corresponding transformation result as transformed output data, and
the output data transform circuit is structured based on the parameter of the convolution operation and a type of an algorithm that transforms data to reduce a computational quantity of the convolution operation.

### Claim 10 (depends on 9)

The memory device of claim 9, wherein the output transform matrix is determined based on the parameter of the convolution operation and the type of the algorithm, and
the output data transform circuit is structured to correspond to the output transform matrix.

### Claim 11 (depends on 1)

The memory device of claim 1, wherein the parameter of the convolution operation is determined based on any one or any combination of any two or more of a size of output data, a size of filter data, a size of input data, a stride interval, and a padding size.

### Claim 12 (depends on 1)

The memory device of claim 1, wherein the function processor comprises either one or both of an adder and a shifter.

### Claim 13 (depends on 1)

The memory device of claim 1, wherein an algorithm that transforms the input data to reduce a computational quantity of the convolution operation is a Winograd algorithm.

### Claim 14 (depends on 1)

The memory device of claim 1, wherein the function processor is configured to output a predetermined value in response to a size of the transformed input data being less than or equal to a threshold.

### Claim 15 (depends on 1)

The memory device of claim 1, wherein the function processor is configured to compute a value of a nonlinear function to determine whether to activate the transformed filter data.

### Claim 16 (depends on 1)

The memory device of claim 1, wherein the function processor comprises an operation processor configured to perform an operation using the transformed input data and the transformed filter data.

### Claim 17 (depends on 16)

The memory device of claim 16, wherein the operation processor comprises a multiplier configured to perform an elementwise multiplication using the transformed input data and the transformed filter data.

### Claim 18 (depends on 1)

The memory device of claim 1, wherein the function processor comprises two or more of following circuit:
an input data transform circuit configured to transform the at least first portion of the input data and output a result of the transforming the at least first portion of the input data;
a filter data transform circuit configured to transform the at least second portion of the filter data and output a result of the transforming the at least second portion of the filter data;
a multiplier configured to perform a multiplication using the transformed input data and the transformed filter data; and
an output data transform circuit configured to transform at least a portion of intermediate output data output by performing the multiplication and output a result of the transforming the at least a portion of intermediate output data.

### Claim 19

A computing apparatus comprising:
a memory configured to store input data and filter data for a convolution operation;
a function processor, embedded inside a memory device, configured to, in response to a read command of at least a portion of data from among the input data and the filter data, transform at least a first portion of the input data using a first transforming matrix and transform at least a second portion of the filter data using a second transforming matrix, respectively during a clock cycle corresponding to the read command, the first transforming matrix and the second transforming matrix being respectively based on a parameter of the convolution operation, and output a corresponding transformation result as transformed data; and
a direct memory access (DMA) processor configured to align and store the at least portion of the data in the memory based on a connection relationship between the memory and the function processor.

### Claim 20 (depends on 19)

The computing apparatus of claim 19, wherein the function processor comprises an input data transform circuit configured to transform the at least first portion of the input data and output a corresponding transformation result as transformed input data, and
the input data transform circuit is structured based on the parameter of the convolution operation and a type of an algorithm applied to transform the at least first portion of the input data to reduce a computational quantity of the convolution operation.

### Claim 21 (depends on 20)

The computing apparatus of claim 20, wherein an input transform matrix is determined based on a combination of the parameter of the convolution operation and the type of the algorithm, and
the input data transform circuit is structured to correspond to the input transform matrix.

### Claim 22 (depends on 19)

The computing apparatus of claim 19, wherein the function processor comprises a filter data transform circuit configured to transform the at least second portion of the filter data and output a corresponding transformation result as transformed filter data, and
the filter data transform circuit is structured based on the parameter of the convolution operation and a type of an algorithm applied to transform the at least second portion of the filter data to reduce a computational quantity of the convolution operation.

### Claim 23 (depends on 19)

The computing apparatus of claim 19, further comprising:
an operation processor configured to perform an operation using the transformed input data and the transformed filter data.

### Claim 24 (depends on 23)

The computing apparatus of claim 23, wherein the operation processor comprises any one or any combination of any two or more of a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a neural processing unit (NPU), and a field programmable gate away (FPGA).

### Claim 25 (depends on 19)

The computing apparatus of claim 19, wherein the function processor is configured to, in response to a write command of at least a portion of intermediate output data that is output through an operation using the transformed input data and the transformed filter data, transform the at least portion of the intermediate output data based on the parameter of the convolution operation during a clock cycle corresponding to the write command and output a corresponding transformation result as transformed intermediate output data.

### Claim 26 (depends on 19)

The computing apparatus of claim 19, wherein the function processor comprises an output data transform circuit configured to transform at least a portion of the intermediate output data and output a corresponding transformation result as transformed intermediate output data, and
the output data transform circuit is structured based on the parameter of the convolution operation and a type of an algorithm that transforms the intermediate output data to reduce a computational quantity of the convolution operation.

### Claim 27

A processor-implemented operation method of a memory device, the method comprising:
receiving at least a portion of data from among input data and filter data from a memory in response to a read command;
transforming, using one or more processor embedded inside the memory device, at least a first portion of the input data using a first transforming matrix and transforming, using the one or more processor, at least a second portion of the filter data using a second transforming matrix, respectively during a clock cycle corresponding to the read command, the first transforming matrix and the second transforming matrix being respectively based on a parameter of a convolution operation, and outputting a corresponding transformation result as transformed data, using a third transforming matrix; and
reduce information loss associated with the outputting by adjusting relationship between the second transforming matrix and the third transforming matrix.

### Claim 28 (depends on 27)

The method of claim 27, wherein the outputting comprises:
determining an input transform matrix based on a combination of the parameter of the convolution operation and a type of an algorithm that transforms the at least first portion of the input data to reduce a computational quantity of the convolution operation; and
transforming the at least first portion of the input data based on the input transform matrix and outputting a corresponding transformation result as transformed input data.

### Claim 29 (depends on 27)

The method of claim 27, wherein the outputting comprises:
determining a filter transform matrix based on a combination of the parameter of the convolution operation and a type of an algorithm that transforms the at least second portion of the filter data to reduce a computational quantity of the convolution operation; and
transforming the at least second portion of the filter data based on the filter transform matrix and outputting a corresponding transformation result as transformed filter data.

### Claim 30 (depends on 27)

The method of claim 27, further comprising:
in response to a write command of at least a portion of intermediate output data that is output through an operation between the transformed input data and the transformed filter data, transforming the at least portion of the intermediate output data based on the parameter of the convolution operation and storing the transformed at least portion of the intermediate output data in the memory during a clock cycle corresponding to the write command.

### Claim 31 (depends on 30)

The method of claim 30, wherein the storing in the memory comprises:
determining an output transform matrix based on both the parameter of the convolution operation and a type of an algorithm that transforms the at least portion of the intermediate output data to reduce a computational quantity of the convolution operation; and
transforming the at least portion of the intermediate output data based on the output transform matrix and storing the same transformed at least portion of the intermediate output data in the memory.

### Claim 32 (depends on 27)

A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of claim 27.

### Claim 33

A computing apparatus comprising:
one or more processors, embedded inside a memory, configured to:
in response to a read command of at least a portion of data among input data and filter data stored in the memory comprising a function-in-memory (FIM), transform at least a first portion of the input data using a first transforming matrix and transform at least a second portion of the filter data using a second transforming matrix, respectively during a clock cycle corresponding to the read command, the first transforming matrix and the second transforming matrix being respectively based on a parameter of the convolution operation, and output a corresponding transformation result as transformed data, using a third transforming matrix; and
reduce information loss associated with the outputting by adjusting relationship between the second transforming matrix and the third transforming matrix.

### Claim 34 (depends on 33)

The computing apparatus of claim 33, wherein the memory is configured to store the input data and the filter data.

### Claim 35 (depends on 34)

The computing apparatus of claim 34, further comprising a direct memory access (DMA) processor configured to align and store the at least portion of the data in the memory based on a connection relationship between the memory and the one or more processors.

### Claim 36 (depends on 33)

The computing apparatus of claim 33, wherein the one or more processors comprise a multiplier accumulator (MAC) configured to perform an operation between the transformed input data and the transformed filter data.