Streaming neural network video codec system and method
AssigneeCisco Technology, Inc.
InventorsMengjun Leng, Samer Lutfi Hijazi, Rafal Pilarczyk, Radhakrishna Shri Venkata Achanta
Life of this patent
Estimating the term…
Abstract
A streaming neural network video codec that leverages temporal redundancy, processing video frames in a rolling window fashion. By encoding video frames using information from multiple frames and only transmitting essential codewords, the system ensures efficient compression with reduced computational overhead. Resiliency to lost codewords is achieved by training with random masks on one or more codewords so that the decoder is robust to packet losses. These techniques achieved improved compression efficiency and significantly reduces the average operations required per frame, allowing for real-time, high-quality video streaming.
Claims20
A method comprising:
- obtaining, from a stream of video frames of video, successive sequences of video frames in a rolling window manner such that there is at least one common video frame between the successive sequences of video frames;
- compressing each successive sequence of video frames with a neural network encoder to produce a codeword sequence for a last frame of respective successive sequences of video frames such that codewords in the codeword sequence for the successive sequences of video frames are the same except for a last codeword of each respective codeword sequence;
- and
- providing to a neural network decoder the last codeword of each of respective successive codeword sequences for each successive sequence of video frames.
A method comprising:
- obtaining, at a neural network decoder, a last codeword of each of respective successive codeword sequences generated by a neural network encoder for successive sequences of video frames of a stream of video frames, each of the successive sequences of video frames having been compressed to produce a codeword sequence for a last frame of respective successive sequences of video frames such that codewords in the codeword sequence for the successive sequences of video frames are the same except for a last codeword of each respective codeword sequence;
- and
- generating, with the neural network decoder, reconstructed video frames from respective last codewords of the respective successive codeword sequences for the successive sequences of video frames.
A method comprising:
- obtaining, from a stream of a video frames {F0, F1, . . . , FT, FT+1, . . . } for video to be encoded, a first sequence of video frames {F0, F1, . . . , FT};
- compressing the first sequence of video frames with a neural network encoder to produce a first codeword sequence {e0, e1, . . . , eK} associated with a last frame FT in the first sequence of video frames;
- obtaining, from the stream of video frames, a next sequence of video frames {F1, . . . , FT, FT+1}, wherein the next sequence of video frames is shifted by a single video frame relative to the first sequence of video frames;
- compressing the next sequence of video frames with the neural network encoder to produce a next codeword sequence {e1, e2, . . . , eK+1} associated with a last frame FT+1 in the next sequence of video frames, wherein the next codeword sequence contains the same codewords as the first codeword sequence except for one codeword eK+1;
- for the first sequence of video frames, providing to a neural network decoder the first codeword sequence;
- and
- for the next sequence of video frames, providing to the neural network decoder the one codeword.
An apparatus comprising:
- an input/output interface configured to obtain, from a stream of video frames of video, successive sequences of video frames in a rolling window manner such that there is at least one common video frame between the successive sequences of video frames;
- and
- a computer processor configured to compress each successive sequence of video frames with a neural network encoder to produce a codeword sequence for a last frame of respective successive sequences of video frames such that codewords in the codeword sequence for the successive sequences of video frames are the same except for a last codeword of each respective codeword sequence, and to provide to a neural network decoder the last codeword of each of respective successive codeword sequences for each successive sequence of video frames.
Dependencies are inferred from the claim text.