Sliding-window rate-distortion optimization in neural network-based video coding

A device for encoding video data, the device comprising:

• a memory configured to store the video data;
• one or more processors implemented in circuitry and configured to:
• encode a set of input frames of the video data with an image analysis neural network to generate a corresponding set of output frames, wherein the set of output frames has an initial amount of distortion relative to the set of input frames;
• determine, for each respective output frame of the set of output frames, a modification that results in a modified amount of total distortion between a sliding window of decoded versions of output frames and a corresponding set of input frames, wherein the respective output frame temporally precedes other output frames in the sliding window;
• update each respective output frame based on the determined modifications, wherein the modified amount of distortion is less than the initial amount of distortion, and wherein for at least one output frame, the update increases an amount of distortion between the at least one output frame and a decoded version of the corresponding input frame;
• and
• output a bitstream that includes the updated output frames.

A non-transitory computer-readable storage medium storing instructions that when executed by one or more processors cause the one or more processors to:

• encode a set of input frames of video data with an image analysis neural network to generate a corresponding set of output frames, wherein the set of output frames has an initial amount of distortion relative to the set of input frames;
• determine, for each respective output frame of the set of output frames, a modification that results in a modified amount of total distortion between a sliding window of decoded versions of output frames and a corresponding set of input frames, wherein the respective output frame temporally precedes other output frames in the sliding window;
• update each respective output frame based on the determined modifications, wherein the modified amount of distortion is less than the initial amount of distortion, and wherein for at least one output frame, the update increases an amount of distortion between the at least one output frame and a decoded version of the corresponding input frame;
• and
• output a bitstream that includes the updated output frames.