Method and system for lossy image or video encoding, transmission and decoding

A method of training one or more models including neural networks, the one or more models being for use in lossy image or video encoding, transmission and decoding, the method comprising the steps of:

• receiving a first input training image;
• encoding the first input training image using a first neural network to produce a latent representation;
• performing a quantization process on the latent representation to produce a quantized latent;
• decoding the quantized latent using a trained denoising model, wherein the initial input to the trained denoising model is a sample from a standard normal conditioned with the upsampled quantized latent, to produce an output image, wherein the output image is an approximation of the first input training image;
• evaluating a loss function based on the rate of the quantized latent;
• evaluating a gradient of the loss function;
• back-propagating the gradient of the loss function through the first neural network to update the parameters of the first neural network;
• repeating the above steps using a first set of training images to produce a first trained neural network.

A data processing system, comprising at least one computer system configured to perform a method for lossy image or video encoding, transmission and decoding, the method comprising the steps of:

• receiving an input image at a first computer system;
• encoding the input image using a first trained neural network to produce a latent representation;
• performing a quantization process on the latent representation to produce a quantized latent;
• transmitting the quantized latent to a second computer system;
• decoding the quantized latent using a trained denoising model, wherein the initial input to the trained denoising model is a sample from a standard normal conditioned with the upsampled quantized latent, to produce an output image;
• wherein the output image is an approximation of the input image.