# US 12,604,021 B2 · Video codec assisted real-time video enhancement using deep learning > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,604,021 B2 - Document ID: US12604021B2 - Status: Granted - Kind: B2 - Application number: 18/404,696 - Filed: 2024-01-04 - Published: 2024-06-27 - Granted: 2026-04-14 - Earliest claimed: 2020-06-26 - Assignees: Intel Corporation - Applicants: Intel Corporation - Inventors: Chen Wang; Ximin Zhang; Huan Dou; Yi-Jen Chiu; Sang-Hee Lee - Web page: https://patentgrep.com/patent/US12604021B2 ## CPC - H04N 19/44: Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder - G06F 18/251: {of input or preprocessed data} - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403) - G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution - G06T 9/002: {using neural networks} - G06V 10/82: using neural networks - H04N 19/102: characterised by the element, parameter or selection affected or controlled by the adaptive coding - H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking - H04N 19/159: Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction - H04N 19/172: the region being a picture, frame or field - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/184: the unit being bits, e.g. of the compressed video stream - H04N 19/30: using hierarchical techniques, e.g. scalability (H04N19/63 takes precedence) - H04N 19/59: involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution - H04N 19/85: using pre-processing or post-processing specially adapted for video compression - G06N 3/045: Combinations of networks - G06T 2207/20084: Artificial neural networks [ANN] - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Techniques related to accelerated video enhancement using deep learning selectively applied based on video codec information are discussed. Such techniques include applying a deep learning video enhancement network selectively to decoded non-skip blocks that are in low quantization parameter frames, bypassing the deep learning network for decoded skip blocks in low quantization parameter frames, and applying non-deep learning video enhancement to high quantization parameter frames. ## Claims ### Claim 1 A method, comprising: determining whether a quantization parameter of a first image is smaller than a threshold, the first image having a first resolution and comprising a first pixel block and a second pixel block; in response to determining that the quantization parameter of the first image is smaller than the threshold: generating, by a deep learning network, a third pixel block from the first pixel block, and retrieving a fourth pixel block from a second image based on the second pixel block, wherein the fourth pixel block is a portion of the second image, and the second image is generated before the first image is generate; and generating a third image by merging the third pixel block and the fourth pixel block, wherein the third image has a second resolution that is higher than the first resolution. ### Claim 2 (depends on 1) The method of claim 1, further comprising: in response to determining that the quantization parameter of the first image is not smaller than the threshold, generating a fifth pixel block by applying interpolation to the first pixel block or the second pixel block, the fifth pixel block having the second resolution. ### Claim 3 (depends on 1) The method of claim 1, wherein retrieving the fourth pixel block from the second image based on the second pixel block comprises: determining a size or position of the fourth pixel block based on a size or position of the second pixel block. ### Claim 4 (depends on 1) The method of claim 1, wherein the third image and the first image have a same time instance in a video. ### Claim 5 (depends on 1) The method of claim 1, wherein the second image is temporally adjacent to the first image in a video. ### Claim 6 (depends on 5) The method of claim 5, further comprising: identifying the second image from the video based on a motion vector of the second pixel block. ### Claim 7 (depends on 1) The method of claim 1, wherein the deep learning network comprises a deep learning super-resolution network. ### Claim 8 One or more non-transitory computer-readable media storing instructions executable to perform operations, the operations comprising: determining whether a quantization parameter of a first image is smaller than a threshold, the first image having a first resolution and comprising a first pixel block and a second pixel block; in response to determining that the quantization parameter of the first image is smaller than the threshold: generating, by a deep learning network, a third pixel block from the first pixel block, and retrieving a fourth pixel block from a second image based on the second pixel block, wherein the fourth pixel block is a portion of the second image, and the second image is generated before the first image is generate; and generating a third image by merging the third pixel block and the fourth pixel block, wherein the third image has a second resolution that is higher than the first resolution. ### Claim 9 (depends on 8) The one or more non-transitory computer-readable media of claim 8, wherein the operations further comprise: in response to determining that the quantization parameter of the first image is not smaller than the threshold, generating a fifth pixel block by applying interpolation to the first pixel block or the second pixel block, the fifth pixel block having the second resolution. ### Claim 10 (depends on 8) The one or more non-transitory computer-readable media of claim 8, wherein retrieving the fourth pixel block from the second image based on the second pixel block comprises: determining a size or position of the fourth pixel block based on a size or position of the second pixel block. ### Claim 11 (depends on 8) The one or more non-transitory computer-readable media of claim 8, wherein the third image and the first image have a same time instance in a video. ### Claim 12 (depends on 8) The one or more non-transitory computer-readable media of claim 8, wherein the second image is temporally adjacent to the first image in a video. ### Claim 13 (depends on 12) The one or more non-transitory computer-readable media of claim 12, wherein the operations further comprise: identifying the second image from the video based on a motion vector of the second pixel block. ### Claim 14 (depends on 8) The one or more non-transitory computer-readable media of claim 8, wherein the deep learning network comprises a deep learning super-resolution network. ### Claim 15 An apparatus, comprising: a computer processor for executing computer program instructions; and a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations comprising: determining whether a quantization parameter of a first image is smaller than a threshold, the first image having a first resolution and comprising a first pixel block and a second pixel block, in response to determining that the quantization parameter of the first image is smaller than the threshold: generating, by a deep learning network, a third pixel block from the first pixel block, and retrieving a fourth pixel block from a second image based on the second pixel block, wherein the fourth pixel block is a portion of the second image, and the second image is generated before the first image is generate, and generating a third image by merging the third pixel block and the fourth pixel block, wherein the third image has a second resolution that is higher than the first resolution. ### Claim 16 (depends on 15) The apparatus of claim 15, wherein the operations further comprise: in response to determining that the quantization parameter of the first image is not smaller than the threshold, generating a fifth pixel block by applying interpolation to the first pixel block or the second pixel block, the fifth pixel block having the second resolution. ### Claim 17 (depends on 15) The apparatus of claim 15, wherein retrieving the fourth pixel block from the second image based on the second pixel block comprises: determining a size or position of the fourth pixel block based on a size or position of the second pixel block. ### Claim 18 (depends on 15) The apparatus of claim 15, wherein the third image and the first image have a same time instance in a video. ### Claim 19 (depends on 15) The apparatus of claim 15, wherein the second image is temporally adjacent to the first image in a video. ### Claim 20 (depends on 19) The apparatus of claim 19, wherein the operations further comprise: identifying the second image from the video based on a motion vector of the second pixel block.