# US 12,010,446 B2 · Image compression using integrated circuit devices having analog inference capability

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,010,446 B2
- Document ID: US12010446B2
- Status: Granted
- Kind: B2
- Application number: 17/940,889
- Filed: 2022-09-08
- Published: 2024-03-14
- Granted: 2024-06-11
- Earliest claimed: 2022-09-08
- Assignees: Micron Technology, Inc.
- Applicants: Micron Technology, Inc.
- Inventors: Poorna Kale
- Web page: https://patentgrep.com/patent/US12010446B2

## CPC

- H04N 25/75: Circuitry for providing, modifying or processing image signals from the pixel array
- G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00)
- G06T 9/002: {using neural networks}
- H04N 25/40: Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method in an integrated circuit device to compress images, including: generating, by an image processing logic circuit and based on first data representative of an input image, input data; generating, by an inference logic circuit and based on the input data, a column of inputs; converting, by the inference logic circuit using voltage drivers connected to wordlines and memory cells storing a weight matrix, and into output currents of the memory cells summed in bitlines, results of bitwise multiplications of bits in the column of inputs and bits stored in the memory cells in a form of threshold voltages of the memory cells; digitizing currents summed in the bitlines to obtain column outputs; generating, by the inference logic circuit, output data based on the column outputs; and generating, using the output data, second data representative of an output image compressed from the input image.

## Claims

### Claim 1

A method, comprising:
generating, by an image processing logic circuit in an integrated circuit device and based on first data representative of an input image, input data for an inference logic circuit in the integrated circuit device;
converting, by the inference logic circuit using voltage drivers connected to wordlines and memory cells in the integrated circuit device results of bitwise multiplications of bits in a column of inputs;
digitizing, by the integrated circuit device, currents summed in the bitlines to obtain column outputs;
generating, by the inference logic circuit, output data based on the column outputs; and
generating, by the image processing logic circuit using the output data, second data representative of an output image compressed from the input image.

### Claim 2 (depends on 1)

The method of claim 1, wherein the input data includes pixel values of the input image and an indication of a location of the weight matrix in a memory cell array in the integrated circuit device.

### Claim 3 (depends on 2)

The method of claim 2, wherein the output data includes coefficients to represent a spatial distribution of the pixel values by a sum of predetermined functions of magnitudes corresponding to the coefficients.

### Claim 4 (depends on 3)

The method of claim 3, wherein the predetermined functions are cosine functions.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
performing, by the image processing logic circuit, quantization of the coefficients; and
encoding, by the image processing logic circuit, a result of the quantization to generate the second data.

### Claim 6 (depends on 2)

The method of claim 2, wherein the weight matrix includes a color matrix; the pixel values are in a first color space; and the output data includes pixel values in a second color space.

### Claim 7 (depends on 2)

The method of claim 2, wherein the output data includes an identification of a segment, in the input image, predicted by an artificial neural network to be of interest;
and the method further comprises:
compressing the segment at a first compression ratio; and
compressing a region of the input image outside of the segment at a second compression ratio higher than the first compression ratio.

### Claim 8 (depends on 2)

The method of claim 2, wherein the output data includes levels of interest in segments of the input image ranked by an artificial neural network; and the method further comprises:
compressing the segments of the input images at compression ratios according to the levels of interest.

### Claim 9 (depends on 2)

The method of claim 2, wherein the output data includes compression ratios predicted by an artificial neural network for segments of the input image.

### Claim 10 (depends on 2)

The method of claim 2, wherein the output data includes a compressed image predicted by an artificial neural network.

### Claim 11

A device, comprising:
an image processing logic circuit;
an inference logic circuit;
voltage drivers;
current digitizers;
an array of memory cells having wordlines and bitlines; and
an integrated circuit package enclosing the memory cells, the voltage drivers, the current digitizers, the inference logic circuit, and the image processing logic circuit;
wherein the image processing logic circuit is configured to use the inference logic circuit to perform operations of multiplication and accumulation to compress first data representative of an input image;
wherein the inference logic circuit is configured to:
receive, from the image processing logic circuit, input data based on the input image;
convert, using the voltage drivers connected to the wordlines and the memory cells results of bitwise multiplications of bits in a column of inputs;
digitize, by the current digitizers connected to the bitlines, currents in the bitlines to obtain column outputs; and
generate, by the inference logic circuit, output data based on the column outputs; and
wherein the image processing logic circuit is configured to use the output data to generate second data representative of an output image compressed from the input image.

### Claim 12 (depends on 11)

The device of claim 11, wherein the inference logic circuit is configured to generate the output data to identify:
coefficients to represent a spatial distribution of the pixel values by a sum of predetermined functions of magnitudes corresponding to the coefficients;
pixel values in a second color space different from a first color space used by the input image;
an identification of a segment in the input image and predicted by an artificial neural network to be of interest;
levels of interest in segments of the input image ranked by an artificial neural network;
compression ratios predicted by an artificial neural network for segments of the input image; or
a compressed image predicted by an artificial neural network for the input image.

### Claim 13 (depends on 12)

The device of claim 12, wherein the inference logic circuit further comprises a plurality of parallel logic circuits each configured to evaluate an activation function of an artificial neuron.

### Claim 14 (depends on 12)

The device of claim 12, further comprising:
an interface operable for a host system to read data from and write data into the memory cells; and
an image sensing pixel array configured on a first integrated circuit die;
wherein the inference logic circuit and the image processing logic circuit are configured on a second integrated circuit die; and
wherein the memory cells are configured on a third integrated circuit die.

### Claim 15 (depends on 12)

The device of claim 12, wherein the memory cells are configured in a plurality of layers of an integrated circuit die.

### Claim 16 (depends on 15)

The device of claim 15, wherein each respective memory cell in the memory cells is configured to output:
a predetermined amount of current in response to a predetermined read voltage when the respective memory cell has a threshold voltage programmed to represent a value of one; or
a negligible amount of current in response to the predetermined read voltage when the threshold voltage is programmed to represent a value of zero.

### Claim 17

An apparatus, comprising:
a first integrated circuit die having an image sensing pixel array;
a second integrated circuit die having an image processing logic circuit, and an inference logic circuit;
a third integrated circuit die having a plurality of layers, each containing an array of memory cells having wordlines and bitlines;
an integrated circuit package configured to enclose the first integrated circuit die, the second integrated circuit die and the third integrated circuit die;
an interface;
voltage drivers; and
current digitizers;
wherein the image processing logic circuit is configured to receive, from the image sensing pixel array, first data representative of an input image;
wherein the inference logic circuit is configured to:
receive, from the image processing logic circuit, input data based on the input image;
convert, using the voltage drivers connected to the wordlines and the memory cells results of bitwise multiplications of bits in a column of inputs;
digitize, by the current digitizers connected to the bitlines, currents in the bitlines to obtain column outputs; and
generate, by the inference logic circuit, output data based on the column outputs; and
wherein the image processing logic circuit is configured to use the output data to generate second data representative of an output image compressed from the input image.

### Claim 18 (depends on 17)

The apparatus of claim 17, wherein the image processing logic circuit is configured to compress segments of the input image at compression ratios that are based on the output data.

### Claim 19 (depends on 18)

The apparatus of claim 18, wherein the output data includes coefficients to represent a spatial distribution of the pixel values by a sum of predetermined functions of magnitudes corresponding to the coefficients; and the image processing logic circuit is configured to:
perform quantization of the coefficients; and
encode a result of the quantization to generate the second data.

### Claim 20 (depends on 17)

The apparatus of claim 17, wherein the output data includes a compressed image predicted by an artificial neural network for the input image.