Hybrid motion-compensated neural network with side-information based video coding
AssigneeGoogle Inc.
InventorsDebargha Mukherjee, Urvang Joshi, Yue Chen, Sarah Parker
Life of this patent
Estimating the term…
Abstract
A hybrid apparatus for coding a video stream includes a first encoder. The first encoder includes a neural network having at least one hidden layer, and the neural network receives source data from the video stream at a first hidden layer of the at least one hidden layer, receives side information correlated with the source data at the first hidden layer, and generates guided information using the source data and the side information. The first encoder outputs the guided information and the side information for a decoder to reconstruct the source data.
Claims20
A hybrid apparatus for coding a video stream, comprising:
- a first encoder that receives source data from the video stream and receives side information correlated with the source date, the first encoder comprising:
- a neural network having at least one hidden layer, wherein the neural network:
- receives the source data at a first hidden layer of the at least one hidden layer;
- receives the side information at the first hidden layer;
- and
- generates guided information using the source data and the side information;
- and
- an entropy coder that entropy encodes block data from the source data into a compressed bitstream, wherein:
- the first encoder outputs the compressed bitstream from the entropy coder to a decoder,
- the neural network outputs the guided information, and
- the first encoder, in addition to providing the side information to the neural network, outputs the side information separately from the compressed bitstream so that the side information bypasses the neural network for the decoder to reconstruct the source data from the compressed bitstream and the side information to produce reconstructed source data.
A method for coding a video stream, comprising:
- providing source data from the video stream to a first encoder that includes an entropy coder and a neural network;
- generating, using the source data, side information;
- inputting the side information to the neural network for encoding the source data to form encoded source data that includes at least block residuals;
- entropy encoding, using the entropy coder, the encoded source data into a compressed bitstream;
- transmitting the compressed bitstream from the entropy coder of the first encoder to a decoder or to storage;
- and
- transmitting the side information from the first encoder to the decoder or to the storage, wherein the side information is transmitted separately from the compressed bitstream without modification by the neural network such that the side information transmitted from the first encoder is unchanged from the side information that is input to the neural network of the first encoder.
A hybrid apparatus for coding a video stream, comprising:
- a first encoder and a first decoder comprising a neural network having a plurality of hidden layers, wherein the neural network:
- receives source data from the video stream at a first hidden layer of the first encoder;
- receives side information correlated with the source data at the first hidden layer of the first encoder;
- generates guided information using the source data and the side information;
- and
- receives the guided information and the side information at a first hidden layer of the first decoder for reconstruction of the source data, and wherein:
- the first encoder comprises an entropy coder to entropy encode at least block residuals of the source data into a compressed bitstream,
- the first encoder transmits the compressed bitstream to the second decoder for reconstruction of the source data, and
- the first encoder outputs the side information separately from the compressed bitstream to the first decoder so that the side information, in addition to being received by the neural network, is also transmitted to the first decoder unchanged by the neural network.
Dependencies are inferred from the claim text.