# US 2022/0286682 A1 · Image encoding and decoding, video encoding and decoding: methods, systems and training methods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0286682 A1
- Document ID: US20220286682A1
- Status: Published application
- Kind: A1
- Application number: 17/748,502
- Filed: 2022-05-19
- Published: 2022-09-08
- Earliest claimed: 2020-10-23
- Assignees: Deep Render Ltd.
- Applicants: Deep Render Ltd
- Inventors: Chri BESENBRUCH; Aleksandar CHERGANSKI; Christopher FINLAY; Alexander LYTCHIER; Jonathan RAYNER; Tom RYDER; Jan XU; Arsalan ZAFAR
- Web page: https://patentgrep.com/patent/US20220286682A1

## CPC

- H04N 19/13: Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]
- H04N 19/59: involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06V 10/422: for representing the structure of the pattern or shape of an object therefor
- H04N 19/124: Quantisation
- H04N 19/42: characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation (H04N19/635 takes precedence)
- H04N 19/91: Entropy coding, e.g. variable length coding [VLC] or arithmetic coding
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Lossy or lossless compression and transmission, comprising the steps of: (i) receiving an input image; (ii) encoding it using an encoder trained neural network, to produce a y latent representation; (iii) encoding the y latent representation using a hyperencoder trained neural network, to produce a z hyperlatent representation; (iv) quantizing the z hyperlatent representation using a predetermined entropy parameter to produce a quantized z hyperlatent representation; (v) entropy encoding the quantized z hyperlatent representation into a first bitstream, using predetermined entropy parameters; (vi) processing the quantized z hyperlatent representation using a hyperdecoder trained neural network to obtain a location entropy parameter μy, an entropy scale parameter σy, and a context matrix Ay of the y latent representation; (vii) processing the y latent representation, the location entropy parameter μy and the context matrix Ay, to obtain quantized latent residuals; (viii) entropy encoding the quantized latent residuals into a second bitstream, using the entropy scale parameter σy; and (ix) transmitting the bitstreams.

## Claims

### Claim 1

A computer implemented method of training a total neural network including a neural network which modifies encoder-decoder performance, the total neural network parameterized by parameters θ and including encoder and decoder neural networks for use in lossy image or video compression, transmission and decoding, the method including the steps of:
(i) receiving an input training image x;
(ii) encoding the input training image using the encoder neural network, to produce a latent representation;
(iii) quantizing the latent representation to produce a quantized latent ŷ;
(iv) using the decoder neural network to produce an output image {circumflex over (x)} from the quantized latent, wherein the output image is an approximation of the input image;
(v) evaluating a Lagrangian loss function including a Lagrange multiplier λ, based on evaluation of a distortion function of differences between the output image and the input training image, and based on differences between a compression rate function of the quantized latent R(ŷ) and a target compression rate r0;
(vi) calculating total neural network parameter gradients using backpropagation of the evaluated Lagrangian loss function;
(vii) updating the total network parameters θ by performing a descent step of the Lagrangian loss function with respect to the total network parameters θ based on the total neural network parameter gradients evaluated in step (vi);
(viii) updating the Lagrange multiplier by performing an ascent step of the Lagrangian loss function with respect to the Lagrange multiplier;
(ix) repeating steps (i) to (viii) using a set of training images, to produce a trained set of parameters θ, and optionally a target compression rate r0; and
(x) storing the trained set of parameters θ.

### Claim 2

A computer implemented method of training a total neural network including a neural network which modifies encoder-decoder performance, the total neural network parameterized by parameters θ and including encoder and decoder neural networks for use in lossy image or video compression, transmission and decoding, the method including the steps of:
(i) receiving an input training image x;
(ii) encoding the input training image using the encoder neural network, to produce a latent representation;
(iii) quantizing the latent representation to produce a quantized latent ŷ;
(iv) using the decoder neural network to produce an output image {circumflex over (x)} from the quantized latent, wherein the output image is an approximation of the input image;
(v) evaluating an augmented Lagrangian loss function including a Lagrange multiplier λ, based on evaluation of a distortion function of differences between the output image and the input training image, and based on differences between a compression rate function of the quantized latent R(ŷ) and a target compression rate r0;
(vi) calculating total neural network parameter gradients using backpropagation of the evaluated augmented Lagrangian loss function;
(vii) updating the total network parameters θ by performing an optimizer step in which a SGD or SGD-like optimizer optimizes the total network parameters θ based on a learning rate for the network parameters optimizer and the total neural network parameter gradients evaluated in step (vi);
(viii) evaluating gradients for the Lagrange multiplier by evaluating an augmented Lagrangian loss function quadratic term weight μ multiplied by a difference between the compression rate function of the quantized latent and the target compression rate;
(ix) updating the Lagrange multiplier by performing an optimizer step in which a SGD or SGD-like optimizer optimizes the Lagrange multiplier based on a learning rate for the Lagrange multiplier optimizer;
(x) repeating steps (i) to (ix) using a set of training images, to produce a trained set of parameters θ, and
(xi) storing the trained set of parameters θ.

### Claim 3 (depends on 2)

The method of claim 2, including the step of clipping the gradients for the Lagrange multiplier evaluated in step (viii), and including updating the Lagrange multiplier by performing the optimizer step in which the SGD or SGD-like optimizer optimizes the Lagrange multiplier based on the learning rate for the Lagrange multiplier optimizer and the clipped gradients for the Lagrange multiplier.

### Claim 4 (depends on 1)

The method of claim 1, wherein the trained set of parameters θ is a converged set of parameters θ.

### Claim 5 (depends on 1)

The method of claim 1, wherein the augmented Lagrangian loss function is D(x, {circumflex over (x)})+λ(R(ŷ)−r0)+μ(R(ŷ)−r0)2/2, where D is a function measuring distortion of data reconstruction.

### Claim 6 (depends on 2)

The method of claim 2, wherein step (ix) includes modifying the Lagrange multiplier by the learning rate for the Lagrange multiplier optimizer times the loss function quadratic term weight multiplied by a difference between the compression rate function of the quantized latent and the target compression rate.

### Claim 7 (depends on 2)

The method of claim 2, wherein decoupling the learning rate for the Lagrange multiplier from the loss function quadratic term weight through the introduction of the factor of the learning rate for the Lagrange multiplier optimizer means that the loss function quadratic term weight can be kept small, while the multiplier converges in a reasonable amount of time.

### Claim 8 (depends on 2)

The method of claim 2, wherein the SGD-like optimizer optimizing the Lagrange multiplier is an Adam optimizer.

### Claim 9 (depends on 2)

The method of claim 2, in which the quantized latent compression rate is calculated using a training quantisation function in the forward pass steps (iii) and (iv), and in the backward pass steps (v) to (viii) (i.e. in gradient computation) when updating the total neural network parameters θ, but the quantized latent compression rate is calculated using an inference quantisation function when performing updates to the Augmented Lagrangian Method's Lagrange multiplier steps (viii), and (ix).

### Claim 10 (depends on 2)

The method of claim 2, in which the quantized latent is calculated using an inference quantisation function in the forward pass steps (iii) and (iv), and in the backward pass steps (v) to (viii) (i.e. in gradient computation) the quantized latent is calculated using a training quantisation function.

### Claim 11 (depends on 2)

The method of claim 2, wherein the set of training images used is a set of training images used to train the encoder and the decoder, modified so as to be compressed to the target compression rate; the distortion function in step (v) is reduced by a scale factor s (e.g. approximately two, initially); during training, a running average of scaled distortion is updated; at predefined iterations of training, the scale factor s is adjusted in proportion to (e.g. two times) the running average, and the Lagrange multiplier is modified by a factor in inverse proportion to the adjustment to the scale factor.

### Claim 12 (depends on 1)

The method of claim 1, wherein the quantized latent is represented using a probability distribution of the latent space, the probability distribution including a location parameter and a scale parameter σ, wherein σ is reduced as the computer implemented method of training proceeds, e.g. using a decaying scale threshold.

### Claim 13 (depends on 12)

The method of claim 12, wherein σ is reduced as the computer implemented method of training proceeds, until σ reaches a final value.

### Claim 14 (depends on 12)

The method of claim 12, wherein σ is reduced as the computer implemented method of training proceeds, using a progressively decreasing thresholding value t.

### Claim 15 (depends on 14)

The method of claim 14, wherein σ is reduced as the computer implemented method of training proceeds, decaying t linearly with respect to the number of training iterations.

### Claim 16 (depends on 14)

The method of claim 14, wherein σ is reduced as the computer implemented method of training proceeds, decaying t exponentially with respect to the number of training iterations.

### Claim 17 (depends on 14)

The method of claim 14, wherein σ is reduced as the computer implemented method of training proceeds, decaying t with respect to the loss metric.

### Claim 18 (depends on 1)

The method of claim 1, wherein the quantized latent is represented using a probability distribution of the latent space, the probability distribution including a location parameter and a scale parameter σ, wherein all realisations of σ are thresholded to a fixed value.

### Claim 19 (depends on 12)

The method of claim 12, wherein all realisations of σ are mapped with functions of a strictly positive codomain, for a defined domain, such as the softplus operation, or the squaring operation with thresholding, or the absolute value operation with thresholding.

### Claim 20

A computer program product executable on a processor to train a total neural network including a neural network which modifies encoder-decoder performance, the total neural network parameterized by parameters θ and including encoder and decoder neural networks for use in lossy image or video compression, transmission and decoding, the computer program product executable on the processor to:
(i) receive an input training image x;
(ii) encode the input training image using the encoder neural network, to produce a latent representation;
(iii) quantize the latent representation to produce a quantized latent ŷ;
(iv) use the decoder neural network to produce an output image {circumflex over (x)} from the quantized latent, wherein the output image is an approximation of the input image;
(v) evaluate a Lagrangian loss function including a Lagrange multiplier λ, based on evaluation of a distortion function of differences between the output image and the input training image, and based on differences between a compression rate function of the quantized latent R(ŷ) and a target compression rate r0;
(vi) calculate total neural network parameter gradients using backpropagation of the evaluated Lagrangian loss function;
(vii) update the total network parameters θ by performing a descent step of the Lagrangian loss function with respect to the total network parameters θ based on the total neural network parameter gradients evaluated in (vi);
(viii) update the Lagrange multiplier by performing an ascent step of the Lagrangian loss function with respect to the Lagrange multiplier;
(ix) repeat (i) to (viii) using a set of training images, to produce a trained set of parameters θ, and optionally a target compression rate r0; and
(x) store the trained set of parameters θ.
