# US 2023/0419555 A1 · Channel-wise autoregressive entropy models for image compression

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0419555 A1
- Document ID: US20230419555A1
- Status: Published application
- Kind: A1
- Application number: 18/461,292
- Filed: 2023-09-05
- Published: 2023-12-28
- Earliest claimed: 2020-09-15
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: David Charles Minnen; Saurabh Singh
- Web page: https://patentgrep.com/patent/US20230419555A1

## CPC

- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06T 9/002: {using neural networks}
- G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)}
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/09: Supervised learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for channel-wise autoregressive entropy models. In one aspect, a method includes processing data using a first encoder neural network to generate a latent representation of the data. The latent representation of data is processed by a quantizer and a second encoder neural network to generate a quantized latent representation of data and a latent representation of an entropy model. The latent representation of data is further processed into a plurality of slices of quantized latent representations of data wherein the slices are arranged in an ordinal sequence. A hyperprior processing network generates a hyperprior parameters and a compressed representation of the hyperprior parameters. For each slice, a corresponding compressed representation is generated using a corresponding slice processing network wherein a combination of the compressed representations form a compressed representation of the data.

## Claims

### Claim 1

A computer-implemented method, comprising:
processing data using a first encoder neural network to generate a latent representation of the data;
processing the latent representation of data, comprising:
processing the latent representation of data by a first quantizer to generate a quantized latent representation of data; and
processing the latent representation of the data using a second encoder neural network to generate a latent representation of an entropy model, wherein the entropy model is defined by one or more probability distribution parameters characterizing one or more code symbol probability distributions;
processing the quantized latent representation of data into a plurality of slices of quantized latent representations of data, wherein each slice of quantized latent representation of data is different from each other slice of quantized latent representation of data, and the slices are arranged in an ordinal sequence;
generating, by a hyperprior processing network and from the latent representation of the entropy model, hyperprior parameters representing the probability distribution of the entropy model and a compressed representation of the hyperprior parameters;
providing the hyperprior parameters as input to a plurality of slice processing networks, wherein each slice processing network also receives as input one of the slices of quantized latent representations of data and generates a compressed representation of the slice that is received as input based at least on the slice as received as input and the hyperprior parameters;
wherein a combination of the compressed representation of the slices received as input form a compressed representation of the data.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating, by the hyperprior processing network and from the latent representation of the entropy model, the hyperprior parameters comprises:
processing the latent representation of the entropy model using a second quantizer to generate a quantized representation of the latent representation of the entropy model;
processing the quantized representation of the latent representation of the entropy model using an encoder and based on a predetermined entropy model that specifies one or more predetermined probability distributions to generate a compressed representation of the hyperprior parameters;
processing the compressed representation of the hyperprior parameters using a decoder to generate a decoded representations of two or more hyperprior parameters;
processing each of the decoded representation of two or more hyperprior parameters using a corresponding decoder neural network to generate the two or more hyperprior parameters.

### Claim 3 (depends on 1)

The method of claim 1, wherein the data comprises an image.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first and the second encoder neural network comprises convolution neural network layers.

### Claim 5 (depends on 1)

The method of claim 1, wherein:
a first slice processing network of the plurality of slice processing networks generates a first augmented slice that represents the first slice of quantized latent representation of data and a latent residual prediction that is a prediction of a residual encoding and decoding error based on the hyperprior parameters and provides the first augmented slice to a subsequent slice processing network.

### Claim 6 (depends on 5)

The method of claim 5, wherein:
each slice processing networks subsequent to the first slice processing network receives a respective subsequent augmented slice from a prior slice processing network and generates a respective subsequent augmented slice that represents the respective subsequent slice of quantized latent representation of data and a respective latent residual prediction that is a prediction of a residual encoding and decoding error based on the hyperprior parameters and each prior subsequent augmented slice.

### Claim 7

A system, comprising:
a data processing apparatus; and
a non-transitory computer readable medium in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising:
processing data using a first encoder neural network to generate a latent representation of the data;
processing the latent representation of data, comprising:
processing the latent representation of data by a first quantizer to generate a quantized latent representation of data; and
processing the latent representation of the data using a second encoder neural network to generate a latent representation of an entropy model, wherein the entropy model is defined by one or more probability distribution parameters characterizing one or more code symbol probability distributions;
processing the quantized latent representation of data into a plurality of slices of quantized latent representations of data, wherein each slice of quantized latent representation of data is different from each other slice of quantized latent representation of data, and the slices are arranged in an ordinal sequence;
generating, by a hyperprior processing network and from the latent representation of the entropy model, hyperprior parameters representing the probability distribution of the entropy model and a compressed representation of the hyperprior parameters;
providing the hyperprior parameters as input to a plurality of slice processing networks, wherein each slice processing network also receives as input one of the slices of quantized latent representations of data and generates a compressed representation of the slice that is received as input based at least on the slice as received as input and the hyperprior parameters;
wherein a combination of the compressed representation of the slices received as input form a compressed representation of the data.

### Claim 8 (depends on 7)

The system of claim 7, wherein generating, by the hyperprior processing network and from the latent representation of the entropy model, the hyperprior parameters comprises:
processing the latent representation of the entropy model using a second quantizer to generate a quantized representation of the latent representation of the entropy model;
processing the quantized representation of the latent representation of the entropy model using an encoder and based on a predetermined entropy model that specifies one or more predetermined probability distributions to generate a compressed representation of the hyperprior parameters;
processing the compressed representation of the hyperprior parameters using a decoder to generate a decoded representations of two or more hyperprior parameters;
processing each of the decoded representation of two or more hyperprior parameters using a corresponding decoder neural network to generate the two or more hyperprior parameters.

### Claim 9 (depends on 7)

The system of claim 7, wherein the data comprises an image.

### Claim 10 (depends on 7)

The system of claim 7, wherein the first and the second encoder neural network comprises convolution neural network layers.

### Claim 11 (depends on 10)

The system of claim 10, wherein:
a first slice processing network of the plurality of slice processing networks generates a first augmented slice that represents the first slice of quantized latent representation of data and a latent residual prediction that is a prediction of a residual encoding and decoding error based on the hyperprior parameters and provides the first augmented slice to a subsequent slice processing network.

### Claim 12 (depends on 11)

The system of claim 11, wherein:
each slice processing networks subsequent to the first slice processing network receives a respective subsequent augmented slice from a prior slice processing network and generates a respective subsequent augmented slice that represents the respective subsequent slice of quantized latent representation of data and a respective latent residual prediction that is a prediction of a residual encoding and decoding error based on the hyperprior parameters and each prior subsequent augmented slice.

### Claim 13

A non-transitory computer readable medium storing instructions that, when executed by one or more data processing apparatus, cause the one or more data processing apparatus to perform operations comprising:
processing data using a first encoder neural network to generate a latent representation of the data;
processing the latent representation of data, comprising:
processing the latent representation of data by a first quantizer to generate a quantized latent representation of data; and
processing the latent representation of the data using a second encoder neural network to generate a latent representation of an entropy model, wherein the entropy model is defined by one or more probability distribution parameters characterizing one or more code symbol probability distributions;
processing the quantized latent representation of data into a plurality of slices of quantized latent representations of data, wherein each slice of quantized latent representation of data is different from each other slice of quantized latent representation of data, and the slices are arranged in an ordinal sequence;
generating, by a hyperprior processing network and from the latent representation of the entropy model, hyperprior parameters representing the probability distribution of the entropy model and a compressed representation of the hyperprior parameters;
providing the hyperprior parameters as input to a plurality of slice processing networks, wherein each slice processing network also receives as input one of the slices of quantized latent representations of data and generates a compressed representation of the slice that is received as input based at least on the slice as received as input and the hyperprior parameters;
wherein a combination of the compressed representation of the slices received as input form a compressed representation of the data.

### Claim 14 (depends on 13)

The non-transitory computer readable medium of claim 13, wherein generating, by the hyperprior processing network and from the latent representation of the entropy model, the hyperprior parameters comprises:
processing the latent representation of the entropy model using a second quantizer to generate a quantized representation of the latent representation of the entropy model;
processing the quantized representation of the latent representation of the entropy model using an encoder and based on a predetermined entropy model that specifies one or more predetermined probability distributions to generate a compressed representation of the hyperprior parameters;
processing the compressed representation of the hyperprior parameters using a decoder to generate a decoded representations of two or more hyperprior parameters;
processing each of the decoded representation of two or more hyperprior parameters using a corresponding decoder neural network to generate the two or more hyperprior parameters.

### Claim 15 (depends on 13)

The non-transitory computer readable medium of claim 13, wherein the data comprises an image.

### Claim 16 (depends on 13)

The non-transitory computer readable medium of claim 13, wherein the first and the second encoder neural network comprises convolution neural network layers.

### Claim 17 (depends on 16)

The non-transitory computer readable medium of claim 16, wherein:
a first slice processing network of the plurality of slice processing networks generates a first augmented slice that represents the first slice of quantized latent representation of data and a latent residual prediction that is a prediction of a residual encoding and decoding error based on the hyperprior parameters and provides the first augmented slice to a subsequent slice processing network.

### Claim 18 (depends on 17)

The non-transitory computer readable medium of claim 17, wherein:
each slice processing networks subsequent to the first slice processing network receives a respective subsequent augmented slice from a prior slice processing network and generates a respective subsequent augmented slice that represents the respective subsequent slice of quantized latent representation of data and a respective latent residual prediction that is a prediction of a residual encoding and decoding error based on the hyperprior parameters and each prior subsequent augmented slice.
