# US 11,909,956 B2 · DNN-based cross component prediction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,909,956 B2 - Document ID: US11909956B2 - Status: Granted - Kind: B2 - Application number: 17/749,730 - Filed: 2022-05-20 - Published: 2022-12-15 - Granted: 2024-02-20 - Earliest claimed: 2021-06-15 - Assignees: Tencent America LLC - Applicants: Tencent America LLC - Inventors: Sheng Lin; Wei Jiang; Wei Wang; Liqiang Wang; Shan Liu; Xiaozhong Xu - Web page: https://patentgrep.com/patent/US11909956B2 ## CPC - H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction - H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/186: the unit being a colour or a chrominance component - H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) - H04N 19/50: using predictive coding (H04N19/61 takes precedence) - H04N 19/59: involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution - G06N 3/08: Learning methods - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Systems and methods for deep neural network (DNN)-based cross component prediction are provided. A method includes inputting a reconstructed luma block of an image or video into a DNN; and predicting, by the DNN, a reconstructed chroma block of the image or video based on the reconstructed luma block that is input. Luma and chroma reference information and side information may also be input into the DNN to predict the reconstructed chroma block. The various inputs may also be generated using processes such as downsampling and transformation. ## Claims ### Claim 1 A method performed by at least one processor, the method comprising: obtaining a reconstructed luma block of an image or video; inputting the reconstructed luma block into a deep neural network (DNN); obtaining reference components and side information associated with the reconstructed luma block, the side information includes block partition depth information; inputting the reference components and the side information into the DNN; and predicting, by the DNN, a reconstructed chroma block of the image or video based on the reconstructed luma block, the reference components, and the side information. ### Claim 2 (depends on 1) The method of claim 1, wherein the reference components comprise at least one from among an adjacent luma reference of the reconstructed luma block and an adjacent chroma reference of the reconstructed chroma block that is to be predicted, and the predicting further comprises predicting, by the DNN, the reconstructed chroma block based on the reconstructed luma block and the least one from among the adjacent luma reference and the adjacent chroma reference that are input. ### Claim 3 (depends on 1) The method of claim 1, wherein the reference components comprise an adjacent luma reference of the reconstructed luma block and an adjacent chroma reference of the reconstructed chroma block that is to be predicted, and the predicting further comprises predicting, by the DNN, the reconstructed chroma block based on the reconstructed luma block, the adjacent luma reference, and the adjacent chroma reference that are input. ### Claim 4 (depends on 1) The method of claim 1, further comprising generating a feature map based on the side information and concatenating the generated feature map with other image-based feature maps for DNN training. ### Claim 5 (depends on 1) The method of claim 1, wherein the side information includes at least one from among a quantization parameter (QP) value. ### Claim 6 (depends on 1) The method of claim 1, further comprising: computing reconstruction qualities of the reconstructed chroma block that is predicted by the DNN, and of another reconstructed chroma block that is predicted using a different predication mode; and obtaining the image or video using one from among the reconstructed chroma block that is predicted by the DNN and the another reconstructed chroma block that is predicted using the different predication mode, based on a highest one from among the reconstruction qualities computed. ### Claim 7 (depends on 1) The method of claim 1, further comprising: generating inputs of the DNN, wherein the predicting comprises predicting, by the DNN, the reconstructed chroma block of the image or video based on the inputs, wherein the generating comprises: reconstructing a luma block and obtaining adjacent luma references of the luma block; downsampling the luma block to obtain the reconstructed luma block as one of the inputs; downsampling the adjacent luma references of the luma block; and transforming the adjacent luma references, that are downsampled, to have a same size as the luma block that is downsampled, and wherein the inputs of the DNN include the luma block that is downsampled and the adjacent luma references that are transformed. ### Claim 8 (depends on 7) The method of claim 7, wherein the luma block is a 2N by 2M block, and the adjacent luma references comprise a 2N by 2K first luma reference block and a 2K by 2M second luma reference block for reference, wherein N, K, and M are integers, 2N is a width, 2M is a height, 2K is a number of rows or columns in luma reference. ### Claim 9 (depends on 8) The method of claim 8, wherein the reconstructed luma block obtained by downsampling the luma block has a size of N×M, and after downsampling the adjacent luma references, the first luma reference block has a size of N by K, and the second reference luma block has a size of K by M. ### Claim 10 (depends on 7) The method of claim 7, wherein the generating further comprises transforming adjacent chroma references of the reconstructed chroma block to be predicted to have the same size as the luma block that is downsampled, wherein the inputs of the DNN include the luma block that is downsampled, the adjacent luma references that are transformed, and the adjacent chroma references that are transformed. ### Claim 11 A system comprising: at least one memory configured to store computer program code; and at least one processor configured to access the computer program code and operate as instructed by the computer program code, the computer program code comprising: inputting code configured to cause the at least one processor to input a reconstructed luma block of an image or video, reference components, and side information associated with the reconstructed luma block into a deep neural network (DNN) that is implemented by the at least one processor, the side information includes block partition depth information; and predicting code configured to cause the at least one processor to predict, by the DNN, a reconstructed chroma block of the image or video based on the reconstructed luma block, the reference components, and the side information that are input. ### Claim 12 (depends on 11) The system of claim 11, wherein the reference components comprise at least one from among an adjacent luma reference of the reconstructed luma block and an adjacent chroma reference of the reconstructed chroma block that is to be predicted, and the predicting code is further configured to cause the at least one processor to predict, by the DNN, the reconstructed chroma block based on the reconstructed luma block and the least one from among the adjacent luma reference and the adjacent chroma reference that are input. ### Claim 13 (depends on 11) The system of claim 11, wherein the reference components comprise an adjacent luma reference of the reconstructed luma block and an adjacent chroma reference of the reconstructed chroma block that is to be predicted, and the predicting code is further configured to cause the at least one processor to predict, by the DNN, the reconstructed chroma block based on the reconstructed luma block, the adjacent luma reference, and the adjacent chroma reference that are input. ### Claim 14 (depends on 11) The system of claim 11, wherein the computer program code further comprises input generating code that is configured to cause the at least one processor to generate a feature map based on the side information and concatenate the generated feature map with other image-based feature maps for DNN training. ### Claim 15 (depends on 11) The system of claim 11, wherein the side information comprises a quantization parameter (QP) value. ### Claim 16 (depends on 11) The system of claim 11, wherein the computer program code further comprises: reconstruction quality code configured to cause the at least one processor to compute reconstruction qualities of the reconstructed chroma block that is predicted by the DNN, and of another reconstructed chroma block that is predicted using a different predication mode; and image obtaining code configured to cause the at least one processor to obtain the image or video by using one from among the reconstructed chroma block that is predicted by the DNN and the another reconstructed chroma block that is predicted using the different predication mode, based on a highest one from among the reconstruction qualities computed. ### Claim 17 (depends on 11) The system of claim 11, wherein the computer program code further comprises: input generating code configured to cause the at least one processor to generate inputs of the DNN, wherein the predicting code is further configured to cause the at least one processor to predict, by the DNN, the reconstructed chroma block of the image or video based on the inputs, wherein the input generating code is configured to cause the at least one processor to generate the inputs of the DNN by causing the at least one processor to: reconstruct a luma block and obtaining adjacent luma references of the luma block; downsample the luma block to obtain the reconstructed luma block as one of the inputs; downsample the adjacent luma references of the luma block; and transform the adjacent luma references, that are downsampled, to have a same size as the luma block that is downsampled, and wherein the inputs of the DNN comprise the luma block that is downsampled and the adjacent luma references that are transformed. ### Claim 18 (depends on 17) The system of claim 17, wherein the luma block is a 2N by 2M block, and the adjacent luma references comprise a 2N by 2K first luma reference block and a 2K by 2M second luma reference block for reference, wherein N, K, and M are integers, 2N is a width, 2M is a height, 2K is a number of rows or columns in luma reference. ### Claim 19 (depends on 18) The system of claim 18, wherein the reconstructed luma block obtained by downsampling the luma block has a size of N×M, and after downsampling the adjacent luma references, the first luma reference block has a size of N by K, and the second reference luma block has a size of K by M. ### Claim 20 A non-transitory computer-readable medium storing computer code that is configured to, when executed by at least one processor, cause the at least one processor to: implement a deep neural network (DNN); input a reconstructed luma block of an image or video, reference components, and side information associated with the reconstructed luma block into the DNN, the side information includes block partition depth information; and predict, by the DNN, a reconstructed chroma block of the image or video based on the reconstructed luma block, the reference components, and the side information that are input.