Training rate control neural networks through reinforcement learning

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training a rate control neural network for use in encoding sequences of video frames, wherein the rate control neural network is configured to, during encoding of an input sequence of video frames and for each video frame in the sequence, receive input features for the video frame and process the input features to generate a respective score for each of a plurality of possible quantization parameter values, the operations comprising:

• obtaining a training example comprising a training sequence of video frames and a target data measure for the training sequence;
• obtaining a historical performance for the training example;
• generating an encoded representation of the training sequence that comprises a sequence of encoded video frames, comprising, for each video frame in the training sequence:
• generating, using the rate control neural network, a policy output for the video frame that comprises a respective policy score for each of the plurality of possible quantization parameter values;
• selecting, using the policy output, a quantization parameter value from the plurality of possible quantization parameter values;
• and
• encoding the video frame using a video codec in accordance with the selected quantization parameter value;
• determining a current performance for the training example, comprising determining (i) a quality measure that measures a quality of the encoded representation and (ii) a data measure that measures an amount of data required to represent the encoded representation;
• generating a reward value from the current performance and the historical performance, wherein generating a reward value from the current performance and the historical performance comprises:
• setting the reward value to a positive value when a current performance measure is greater than a historical performance measure by more than a first threshold amount;
• and
• training the rate control neural network using the reward value through reinforcement learning.

One or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a rate control neural network for use in encoding sequences of video frames, wherein the rate control neural network is configured to, during encoding of an input sequence of video frames and for each video frame in the sequence, receive input features for the video frame and process the input features to generate a respective score for each of a plurality of possible quantization parameter values, the method comprising:

• obtaining a training example comprising a training sequence of video frames and a target data measure for the training sequence;
• obtaining a historical performance for the training example;
• generating an encoded representation of the training sequence that comprises a sequence of encoded video frames, comprising, for each video frame in the training sequence:
• generating, using the rate control neural network, a policy output for the video frame that comprises a respective policy score for each of the plurality of possible quantization parameter values;
• selecting, using the policy output, a quantization parameter value from the plurality of possible quantization parameter values;
• and
• encoding the video frame using a video codec in accordance with the selected quantization parameter value;
• determining a current performance for the training example, comprising determining (i) a quality measure that measures a quality of the encoded representation and (ii) a data measure that measures an amount of data required to represent the encoded representation;
• generating a reward value from the current performance and the historical performance, wherein generating a reward value from the current performance and the historical performance comprises:
• setting the reward value to a positive value when a current performance measure is greater than a historical performance measure by more than a first threshold amount;
• and
• training the rate control neural network using the reward value through reinforcement learning.