Video compression using recurrent-based machine learning systems

An apparatus for processing video data, comprising:

• a memory;
• and
• a processor coupled to the memory and configured to:
• obtain, using an encoder portion of a neural network system, an input video frame for a current time step of operation of the neural network system, reconstructed motion estimation data from a previous time step of operation of the neural network system, reconstructed residual data from the previous time step of operation of the neural network system, and recurrent state data from at least one recurrent layer of a decoder portion of the neural network system from the previous time step of operation;
• generate, using the encoder portion, output data for the current time step of operation of the neural network system, the output data being generated at least partially based on the input video frame, the reconstructed motion estimation data from the previous time step of operation, the reconstructed residual data from the previous time step of operation, and the recurrent state data from the previous time step of operation;
• obtain, using the decoder portion of a neural network system, the output data for the current time step of operation and the recurrent state data from the at least one recurrent layer of the decoder portion from the previous time step of operation;
• and
• generate, at least partially based on the output data and the recurrent state data from the previous time step of operation, a reconstructed video frame for the current time step of operation, the reconstructed video frame representing the input video frame.

A method of processing video data, the method comprising:

• obtaining, by a decoder portion of a neural network system, output data for a current time step of operation of the neural network system and recurrent state data from at least one recurrent layer of the decoder portion from a previous time step of operation of the neural network system;
• determining, by the decoder portion of the neural network system, motion estimation data for the current time step of operation at least partially based on the output data and the recurrent state data from the previous time step of operation;
• obtaining a previously reconstructed video frame generated during the previous time step of operation;
• generating a warped reconstructed video frame for the current time step of operation at least in part by modifying one or more pixels of the previously reconstructed video frame using the motion estimation data determined for the current time step;
• and
• generating a reconstructed video frame for the current time step of operation at least partially based on the warped reconstructed video frame.

An apparatus for processing video data, comprising:

• a memory;
• and
• a processor coupled to the memory and configured to:
• obtain, using a decoder portion of a neural network system, output data for a current time step of operation of the neural network system and recurrent state data from at least one recurrent layer of the decoder portion from a previous time step of operation of the neural network system;
• determine, using the decoder portion of the neural network system, motion estimation data for the current time step of operation at least partially based on the output data and the recurrent state data from the previous time step of operation;
• obtain a previously reconstructed video frame generated during the previous time step of operation;
• generate a warped reconstructed video frame for the current time step of operation at least in part by modifying one or more pixels of the previously reconstructed video frame using the motion estimation data determined for the current time step;
• and
• generate a reconstructed video frame for the current time step of operation at least partially based on the warped reconstructed video frame.