Image encoding and decoding, video encoding and decoding: methods, systems and training methods

A computer program product stored on a non-transitory computer readable medium executable on a processor to train a total neural network including a neural network which modifies encoder-decoder performance, the total neural network parameterized by parameters θ and including encoder and decoder neural networks for use in lossy image or video compression, transmission and decoding, the computer program product stored on a non-transitory computer readable medium executable on the processor to:

• (i) receive an input training image x;
• (ii) encode the input training image using the encoder neural network, to produce a latent representation;
• (iii) quantize the latent representation to produce a quantized latent ŷ;
• (iv) use the decoder neural network to produce an output image {circumflex over (x)} from the quantized latent, wherein the output image is an approximation of the input image;
• (v) evaluate a Lagrangian loss function including a Lagrange multiplier λ, based on evaluation of a distortion function of differences between the output image and the input training image, and based on differences between a compression rate function of the quantized latent R(ŷ) and a target compression rate r0;
• (vi) calculate total neural network parameter gradients using backpropagation of the evaluated Lagrangian loss function;
• (vii) update the total network parameters θ by performing a descent step of the Lagrangian loss function with respect to the total network parameters θ based on the total neural network parameter gradients evaluated in (vi);
• (viii) update the Lagrange multiplier by performing an ascent step of the Lagrangian loss function with respect to the Lagrange multiplier;
• (ix) repeat (i) to (viii) using a set of training images, to produce a trained set of parameters θ, and optionally a target compression rate r0;
• and
• (x) store the trained set of parameters θ.