# US 2022/0279183 A1 · Image compression and decoding, video compression and decoding: methods and systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0279183 A1
- Document ID: US20220279183A1
- Status: Published application
- Kind: A1
- Application number: 17/740,716
- Filed: 2022-05-10
- Published: 2022-09-01
- Earliest claimed: 2020-04-29
- Assignees: Deep Render Ltd.
- Applicants: Deep Render Ltd
- Inventors: Chri BESENBRUCH; Ciro CURSIO; Christopher FINLAY; Vira KOSHKINA; Alexander LYTCHIER; Jan XU; Arsalan ZAFAR
- Web page: https://patentgrep.com/patent/US20220279183A1

## CPC

- H04N 19/126: Details of normalisation or weighting functions, e.g. normalisation matrices or variable uniform quantisers
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 3/094: Adversarial learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06T 3/4046: using neural networks
- G06T 9/002: {using neural networks}
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- H04N 19/13: Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]
- H04N 19/503: involving temporal prediction (adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109)
- H04N 19/91: Entropy coding, e.g. variable length coding [VLC] or arithmetic coding
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

There is disclosed a computer-implemented method for lossy image or video compression, transmission and decoding, the method including the steps of: (i) receiving an input image at a first computer system; ({umlaut over (υ)}) encoding the input image using a first trained neural network, using the first computer system, to produce a latent representation; (iii) quantizing the latent representation using the first computer system to produce a quantized latent; (iv) entropy encoding the quantized latent into a bitstream, using the first computer system; (v) transmitting the bitstream to a second computer system; (vi) the second computer system entropy decoding the bitstream to produce the quantized latent; (vii) the second computer system using a second trained neural network to produce an output image from the quantized latent, wherein the output image is an approximation of the input image. Related computer-implemented methods, systems, computer-implemented training methods and computer program products are disclosed.

## Claims

### Claim 1

A computer implemented method of training a first neural network and a second neural network, the neural networks being for use in lossy image or video compression, transmission and decoding, the method including the steps of:
(i) receiving an input training image;
(ii) encoding the input training image using the first neural network, to produce a latent representation;
(iii) quantizing the latent representation to produce a quantized latent;
(iv) using the second neural network to produce an output image from the quantized latent, wherein the output image is an approximation of the input image;
(v) evaluating a loss function based on differences between the output image and the input training image;
(vi) evaluating a gradient of the loss function;
(vii) back-propagating the gradient of the loss function through the second neural network and through the first neural network, to update weights of the second neural network and of the first neural network; and
(viii) repeating steps (i) to (vii) using a set of training images, to produce a trained first neural network and a trained second neural network, and
(ix) storing the weights of the trained first neural network and of the trained second neural network;
wherein the loss function is a weighted sum of a rate term and a distortion term,
wherein split quantisation is used during the evaluation of the gradient of the loss function, with a combination of two quantisation proxies for the rate term and the distortion term.

### Claim 2 (depends on 1)

The method of claim 1, wherein during quantization of the latent representation, actual quantisation is replaced by noise quantisation.

### Claim 3 (depends on 2)

The method of claim 2, wherein a noise distribution used for noise quantization is uniform, Gaussian or Laplacian distributed, or a Cauchy distribution, a Logistic distribution, a Student's t distribution, a Gumbel distribution, an Asymmetric Laplace distribution, a skew normal distribution, an exponential power distribution, a Johnson's SU distribution, a generalized normal distribution, or a generalized hyperbolic distribution, or any commonly known univariate or multivariate distribution.

### Claim 4 (depends on 1)

The method of claim 1, wherein an entropy model of a distribution with an unbiased rate loss gradient is used for quantisation of the latent representation.

### Claim 5 (depends on 1)

The method of claim 1, the method further including use of a Laplacian entropy model.

### Claim 6 (depends on 1)

The method of claim 1, wherein noise quantisation is used for the rate term and STE quantisation is used for the distortion term.

### Claim 7 (depends on 6)

The method of claim 6, wherein either of the noise quantisation or the STE quantisation overrides the gradients of the other.

### Claim 8 (depends on 6)

The method of claim 6, wherein the noise quantisation overrides the gradients for the STE quantisation.

### Claim 9 (depends on 1)

The method of claim 1, wherein QuantNet modules are used for learning a differentiable mapping mimicking true quantisation.

### Claim 10 (depends on 1)

The method of claim 1, wherein learned gradient mappings are used for explicitly learning the backward function of a true quantisation operation.

### Claim 11 (depends on 1)

The method of claim 1, wherein discrete density models are used, such as by soft-discretisation of the PDF.

### Claim 12 (depends on 1)

The method of claim 1, wherein context-aware quantisation techniques are used by including flexible parameters in the quantisation function.

### Claim 13 (depends on 1)

The method of claim 1, wherein a parametrisation scheme is used for bin width parameters.

### Claim 14 (depends on 1)

The method of claim 1, wherein context-aware quantisation techniques are used in a transformed latent space, using bijective mappings.

### Claim 15 (depends on 1)

The method of claim 1, the method further including modelling of second-order effects for the minimisation of quantisation errors.

### Claim 16 (depends on 15)

The method of claim 15, further including computing the Hessian matrix of the loss function.
