# US 12,015,776 B2 · Image compression and decoding, video compression and decoding: methods and systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,015,776 B2
- Document ID: US12015776B2
- Status: Granted
- Kind: B2
- Application number: 18/230,314
- Filed: 2023-08-04
- Published: 2023-11-23
- Granted: 2024-06-18
- Earliest claimed: 2020-04-29
- Assignees: Deep Render Ltd.
- Applicants: Deep Render Ltd
- Inventors: Chri Besenbruch; Ciro Cursio; Christopher Finlay; Vira Koshkina; Alexander Lytchier; Jan Xu; Arsalan Zafar
- Web page: https://patentgrep.com/patent/US12015776B2

## CPC

- H04N 19/126: Details of normalisation or weighting functions, e.g. normalisation matrices or variable uniform quantisers
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 3/094: Adversarial learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06T 3/4046: using neural networks
- G06T 9/002: {using neural networks}
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- H04N 19/13: Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]
- H04N 19/503: involving temporal prediction (adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109)
- H04N 19/91: Entropy coding, e.g. variable length coding [VLC] or arithmetic coding
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A computer-implemented method for lossy image or video compression, transmission and decoding, the method including the steps of: (i) receiving an input image at a first computer system; (ii) encoding the input image using a first trained neural network, using the first computer system, to produce a latent representation; (iii) quantizing the latent representation using the first computer system to produce a quantized latent; (iv) entropy encoding the quantized latent into a bitstream, using the first computer system; (v) transmitting the bitstream to a second computer system; (vi) the second computer system entropy decoding the bitstream to produce the quantized latent; (vii) the second computer system using a second trained neural network to produce an output image from the quantized latent, wherein the output image is an approximation of the input image. Related computer-implemented methods, systems, computer-implemented training methods and computer program products.

## Claims

### Claim 1

Computer implemented method of training a first neural network and a second neural network, the neural networks being for use in lossy image or video compression, transmission and decoding, the method including the steps of:
(i) receiving a plurality of input training images associated with a corresponding plurality of views of a scene;
(ii) encoding the plurality of input training images using the first neural network, to produce a latent representation;
(iii) quantizing the latent representation to produce a quantized latent;
(iv) using the second neural network to produce a plurality of output images associated with said corresponding plurality of views from the quantized latent, wherein the plurality of output images are an approximation of the plurality of input images;
(v) evaluating a loss function based on differences between the plurality of output images and the plurality of input images;
(vi) evaluating a gradient of the loss function;
(vii) back-propagating the gradient of the loss function through the second neural network and through the first neural network, to update weights of the second neural network and of the first neural network; and
(viii) repeating steps (i) to (vii) using a set of said plurality of input training images, to produce a trained first neural network and a trained second neural network, and
(ix) storing the weights of the trained first neural network and of the trained second neural network.

### Claim 2 (depends on 1)

The method of claim 1, wherein the loss function is evaluated as a weighted sum of differences between the output images and the input training images, and the estimated bits of the quantized image latents.

### Claim 3 (depends on 1)

The method of claim 1, wherein the steps of the method are performed by a computer system.

### Claim 4 (depends on 1)

The method of claim 1, wherein the loss function includes using a single image depth-map estimation of the plurality of input training images and the plurality of output images and then measuring the distortion between the depth maps of the plurality of input training images and the plurality of output images.

### Claim 5 (depends on 1)

The method of claim 1, wherein the loss function includes using a reprojection into a 3-d coordinate system using said plurality of input training images and a reprojection into the 3-d coordinate system using said plurality of output images and a loss measuring the difference of the resulting 3-d coordinate systems.

### Claim 6 (depends on 1)

The method of claim 1, wherein the loss function includes using optical flow methods that establish correspondence between pixels in said plurality of input training images and said plurality of output images and a loss to minimise one or more flow-maps.

### Claim 7 (depends on 1)

The method of claim 1, wherein positional location information associated with one or more cameras associated with the plurality of input images and their absolute or relative configuration are encoded in the neural networks as a prior through the training process.

### Claim 8 (depends on 1, 6)

A computer-implemented method for lossy image or video compression, transmission and decoding, the method including the steps of:
(i) receiving a plurality of input images associated with a corresponding plurality of views of a scene at a first computer system;
(ii) encoding the plurality of input images using a first trained neural network trained according to the method of claim 6, using the first computer system, to produce a latent representation;
(iii) quantizing the latent representation using the first computer system to produce a quantized latent;
(iv) entropy encoding the quantized latent into a bitstream, using the first computer system;
(v) transmitting the bitstream to a second computer system;
(vi) the second computer system entropy decoding the bitstream to produce the quantized latent;
(vii) the second computer system using a second trained neural network trained according to the method of claim 1, to produce a plurality of output images associated with said plurality of views from the quantized latent, wherein the plurality of output images are an approximation of the plurality of input images.

### Claim 9 (depends on 8)

The method of claim 8, wherein in step (vii) the output images are stored.

### Claim 10 (depends on 8)

The method of claim 8, wherein ground-truth dependencies between the plurality of input images are used as additional input to the first neural network.

### Claim 11 (depends on 8)

The method of claim 8, wherein depth maps associated with the plurality of input images are used as additional input to the first neural network.

### Claim 12 (depends on 8)

The method of claim 8, wherein optical flow data associated with the plurality of input images are used as additional input to the first neural network.