# US 12,192,440 B2 · Neural network based residual coding and prediction for predictive coding > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,192,440 B2 - Document ID: US12192440B2 - Status: Granted - Kind: B2 - Application number: 17/568,266 - Filed: 2022-01-04 - Published: 2022-06-16 - Granted: 2025-01-07 - Earliest claimed: 2019-01-22 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Jiefu Zhai; Xingyu Zhang; Xiaosong Zhou; Jun Xin; Hsi-Jung Wu; Yeping Su - Web page: https://patentgrep.com/patent/US12192440B2 ## CPC - H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction - H04N 19/50: using predictive coding (H04N19/61 takes precedence) - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0499: Feedforward networks - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - H04N 19/103: Selection of coding mode or of prediction mode - H04N 19/12: Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264 - H04N 19/147: according to rate distortion criteria (rate-distortion as a criterion for motion estimation H04N19/567) - H04N 19/159: Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction - H04N 19/172: the region being a picture, frame or field - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/463: by compressing encoding parameters before transmission - H04N 19/60: using transform coding - H04N 19/61: in combination with predictive coding - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Systems and methods disclosed for video compression, utilizing neural networks for predictive video coding. Processes employed combine multiple banks of neural networks with codec system components to carry out the coding and decoding of video data. ## Claims ### Claim 1 A method for coding a video stream, comprising: for a pixel block of an input frame to be coded, generating a pixel block prediction based on input data derived from reference data of previously-coded data of the video stream; generating a residual block representing a difference between the pixel block and the pixel block prediction; coding the residual block; and packing the coded residual block and associated coding parameters in a coded video stream; wherein at least one of the generating a pixel block prediction and the coding the residual block is performed using a plurality of neural networks banks, and wherein different ones of the plurality of neural network banks are trained to operate at a different bitrates from each other. ### Claim 2 (depends on 1) The method of claim 1, wherein each bank in the plurality of neural-network banks produces a corresponding bank output with a corresponding likelihood, and a mixer produces an output for the plurality of neural network banks based on a combination of the bank outputs and their corresponding likelihoods. ### Claim 3 (depends on 2) The method of claim 2, wherein the combination of the bank outputs is a linear combination of the bank outputs weighted by their corresponding likelihoods. ### Claim 4 (depends on 2) The method of claim 2, wherein each bank output is a pixel block prediction, and the likelihoods are based on a prediction error of the corresponding pixel block prediction. ### Claim 5 (depends on 2) The method of claim 2, wherein the associated coding parameters include an indication of which of the plurality of the neural network banks were used by the mixer and their corresponding likelihoods. ### Claim 6 (depends on 1) The method of claim 1, wherein each of the plurality of neural network banks are characterized by different activation functions used in the respective neural network banks. ### Claim 7 (depends on 1) The method of claim 1, wherein each of the plurality of neural network banks are characterized by neural network weights that are trained with different categories of training data from others in the plurality of neural network banks. ### Claim 8 (depends on 1) The method of claim 1, wherein each of the plurality of neural network banks are characterized a different training cost function from others in the plurality of neural network banks. ### Claim 9 (depends on 1) The method of claim 1, wherein the plurality of neural network banks are trained to operate with a different cost function from each other. ### Claim 10 (depends on 1) The method of claim 1, further comprising transmitting the reconstructed pixel block over a communication link to a receiving terminal. ### Claim 11 A method for decoding a coded video stream, comprising: for coded residual block to be decoded, extracting the coded residual block and associated coding parameters from the coded video stream; decoding the coded residual block, based on the coding parameters, to generate a decoded residual block; generating a pixel block prediction based reference data from previously decoded data of the coded video stream; and generating a reconstructed pixel block based on a sum of the decoded residual block and the pixel block prediction; wherein at least one of the generating a pixel block prediction and the decoding the coded residual block is performed using a plurality of neural networks banks, and wherein different ones of the plurality of neural network banks are trained to operate at a different bitrates from each other. ### Claim 12 (depends on 11) The method of claim 11, wherein each bank in the plurality of neural-network banks produces a corresponding bank output with a corresponding likelihood, and a mixer produces an output for the plurality of neural network banks based on a combination of the bank outputs and their corresponding likelihoods. ### Claim 13 (depends on 12) The method of claim 12, wherein the combination of the bank outputs is a linear combination of the bank outputs weighted by their corresponding likelihoods. ### Claim 14 (depends on 12) The method of claim 12, wherein each bank output is a pixel block prediction, and the likelihoods are based on a prediction error of the corresponding pixel block prediction. ### Claim 15 (depends on 12) The method of claim 12, wherein the associated coding parameters include an indication of which of the plurality of the neural network banks were used by the mixer and their corresponding likelihoods. ### Claim 16 (depends on 11) The method of claim 11, wherein each of the plurality of neural network banks are characterized by different activation functions used in the respective neural network banks. ### Claim 17 (depends on 11) The method of claim 11, wherein each of the plurality of neural network banks are characterized by neural network weights that are trained with different categories of training data from others in the plurality of neural network banks. ### Claim 18 (depends on 11) The method of claim 11, wherein each of the plurality of neural network banks are characterized a different training cost function from others in the plurality of neural network banks. ### Claim 19 (depends on 11) The method of claim 11, wherein the plurality of neural network banks are trained to operate with a different cost function from each other. ### Claim 20 (depends on 11) The method of claim 11, generating, based on the reconstructed pixel block based, a reconstructed video stream. ### Claim 21 A computer system for decoding a coded video stream, comprising: at least one processor; at least one memory comprising instructions configured to be executed by the at least one processor to perform a method comprising: for coded residual block to be decoded, extracting the coded residual block and associated coding parameters from the coded video stream; decoding the coded residual block, based on the coding parameters, to generate a decoded residual block; generating a pixel block prediction based reference data from previously decoded data of the coded video stream; and generating a reconstructed pixel block based on a sum of the decoded residual block and the pixel block prediction; wherein at least one of the generating a pixel block prediction and the decoding the coded residual block is performed using a plurality of neural networks banks, and wherein different ones of the plurality of neural network banks are trained to operate at a different bitrates from each other. ### Claim 22 A non-transitory computer-readable storage media for storing computer instructions that, when executed by a processor, cause decoding of a coded video stream, including: for a coded residual block to be decoded, extracting the coded residual block and associated coding parameters from the code video stream; decoding the coded residual block, based on the coding parameters, to generate a decoded residual block; generating a pixel block prediction based reference data from previously decoded of the coded video stream; and generating a reconstructed pixel block based on a sum of the decode residual block and the pixel block prediction; wherein at least one of the generating a pixel block prediction and the decoding the coded residual block is performed using a plurality of neural network banks, and wherein different ones of the plurality of neural network banks are trained to operate at different bitrates from each other.