Back to results
US 12,192,440 B2Granted
Neural network based residual coding and prediction for predictive coding
AssigneeApple Inc.
InventorsJiefu Zhai, Xingyu Zhang, Xiaosong Zhou, Jun Xin, Hsi-Jung Wu, Yeping Su
Life of this patent
Estimating the term…
Abstract
Systems and methods disclosed for video compression, utilizing neural networks for predictive video coding. Processes employed combine multiple banks of neural networks with codec system components to carry out the coding and decoding of video data.
Claims22
A method for coding a video stream, comprising:
- for a pixel block of an input frame to be coded, generating a pixel block prediction based on input data derived from reference data of previously-coded data of the video stream;
- generating a residual block representing a difference between the pixel block and the pixel block prediction;
- coding the residual block;
- and
- packing the coded residual block and associated coding parameters in a coded video stream;
- wherein at least one of the generating a pixel block prediction and the coding the residual block is performed using a plurality of neural networks banks, and
- wherein different ones of the plurality of neural network banks are trained to operate at a different bitrates from each other.
4 more elements
A method for decoding a coded video stream, comprising:
- for coded residual block to be decoded, extracting the coded residual block and associated coding parameters from the coded video stream;
- decoding the coded residual block, based on the coding parameters, to generate a decoded residual block;
- generating a pixel block prediction based reference data from previously decoded data of the coded video stream;
- and
- generating a reconstructed pixel block based on a sum of the decoded residual block and the pixel block prediction;
- wherein at least one of the generating a pixel block prediction and the decoding the coded residual block is performed using a plurality of neural networks banks, and
- wherein different ones of the plurality of neural network banks are trained to operate at a different bitrates from each other.
4 more elements
A computer system for decoding a coded video stream, comprising:
- at least one processor;
- at least one memory comprising instructions configured to be executed by the at least one processor to perform a method comprising:
- for coded residual block to be decoded, extracting the coded residual block and associated coding parameters from the coded video stream;
- decoding the coded residual block, based on the coding parameters, to generate a decoded residual block;
- generating a pixel block prediction based reference data from previously decoded data of the coded video stream;
- and
- generating a reconstructed pixel block based on a sum of the decoded residual block and the pixel block prediction;
- wherein at least one of the generating a pixel block prediction and the decoding the coded residual block is performed using a plurality of neural networks banks, and
- wherein different ones of the plurality of neural network banks are trained to operate at a different bitrates from each other.
6 more elements
A non-transitory computer-readable storage media for storing computer instructions that, when executed by a processor, cause decoding of a coded video stream, including:
- for a coded residual block to be decoded, extracting the coded residual block and associated coding parameters from the code video stream;
- decoding the coded residual block, based on the coding parameters, to generate a decoded residual block;
- generating a pixel block prediction based reference data from previously decoded of the coded video stream;
- and generating a reconstructed pixel block based on a sum of the decode residual block and the pixel block prediction;
- wherein at least one of the generating a pixel block prediction and the decoding the coded residual block is performed using a plurality of neural network banks, and wherein different ones of the plurality of neural network banks are trained to operate at different bitrates from each other.
2 more elements
Dependencies are inferred from the claim text.