Rate control machine learning models with feedback control for video encoding

A system comprising:

• one or more computers;
• and
• one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for encoding a video comprising a sequence of video frames to generate a respective encoded representation of each video frame, the operations comprising:
• for each video frame:
• obtaining a feature embedding for the video frame;
• processing an input comprising the feature embedding for the video frame using a rate control machine learning model to generate a respective score for each of a plurality of possible quantization parameter values;
• selecting a quantization parameter value from the plurality of possible quantization parameter values using the scores;
• and
• processing the video frame using an encoding model, in accordance with a quantization step size associated with the selected quantization parameter value, to generate the encoded representation of the video frame;
• wherein the rate control machine learning model has a plurality of model parameters that are trained on a set of training examples, wherein each training example comprises data defining:
• (i) a respective feature embedding for each training video frame of a training video, and (ii) a respective target quantization parameter value for each training video frame.

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for encoding a video comprising a sequence of video frames to generate a respective encoded representation of each video frame, the operations comprising:

• for each video frame:
• obtaining a feature embedding for the video frame;
• processing an input comprising the feature embedding for the video frame using a rate control machine learning model to generate a respective score for each of a plurality of possible quantization parameter values;
• selecting a quantization parameter value from the plurality of possible quantization parameter values using the scores;
• and
• processing the video frame using an encoding model, in accordance with a quantization step size associated with the selected quantization parameter value, to generate the encoded representation of the video frame;
• wherein the rate control machine learning model has a plurality of model parameters that are trained on a set of training examples, wherein each training example comprises data defining:
• (i) a respective feature embedding for each training video frame of a training video, and (ii) a respective target quantization parameter value for each training video frame.