# US 12,118,692 B2 · Image super-resolution > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,692 B2 - Document ID: US12118692B2 - Status: Granted - Kind: B2 - Application number: 17/544,981 - Filed: 2021-12-08 - Published: 2022-06-23 - Granted: 2024-10-15 - Earliest claimed: 2020-12-17 - Assignees: Picsart, Inc. - Applicants: Picsart, Inc. - Inventors: Victor Kulikov - Web page: https://patentgrep.com/patent/US12118692B2 ## CPC - G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/08: Learning methods - G06N 3/088: Non-supervised learning, e.g. competitive learning - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/09: Supervised learning - G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403) - G06T 3/4046: using neural networks - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Techniques are described for upscaling low-resolution image data. In an embodiment, input image data of a low-resolution image is received to generate output image data for a high-resolution image of the low-resolution image. The input image data is interpolated to match the output size, and, based on the original input image data, the residual image data is determined. The interpolated image data is combined with the residual image data to generate the output image data for the high-resolution image of the output size. The techniques further include training one or more learning models for determining the residual image data based on the original input image data. ## Claims ### Claim 1 A computer-implemented method comprising: receiving input image data of a low-resolution image, having an input size, to generate output image data for a high-resolution image having an output size, the output size being greater than the input size; interpolating the input image data to match the output size, thereby generating interpolated image data for an interpolated image of the output size; determining residual image data based on the input image data, the determining the residual image data based on the input image data comprising: providing the input image data to a first convolutional neural network (CNN) trained to estimate probability density of high-resolution indexes for the input image data to generate estimated probability density of intermediate indexes; based on the estimated probability density of intermediate indexes, selecting a plurality of intermediate indexes; providing the plurality of intermediate indexes as input to a second convolutional neural network to generate the residual image data; combining the interpolated image data with the residual image data to generate the output image data for the high-resolution image of the output size. ### Claim 2 (depends on 1) The method of claim 1, wherein determining the residual image data based on the input image data further comprises: providing the input image data to one or more models to generate the residual image data. ### Claim 3 (depends on 1) The method of claim 1, wherein determining the residual image data based on the input image data further comprises: providing the input image data to one or more models to generate a plurality of intermediate indexes; decoding the plurality of intermediate indexes to generate the residual image data. ### Claim 4 (depends on 1) The method of claim 1, wherein determining the residual image data based on the input image data further comprises: providing the input image data to a probability density estimation model to generate the estimated probability density of intermediate indexes; based on the estimated probability density of intermediate indexes, selecting a plurality of intermediate indexes; decoding the plurality of intermediate indexes to generate the residual image data. ### Claim 5 A computer-implemented method comprising: selecting at least one image data pair from a training dataset, the at least one image data pair of the training dataset including low-resolution image data and high-resolution image data relating to a same particular image, the high-resolution image data having a desired output size; interpolating the low-resolution image data to the desired output size thereby generating interpolated low-resolution image data of the desired output size; based on the high-resolution image data and the interpolated low-resolution image data, generating high-frequency residual image data of the particular image; based on the high-frequency residual image data of the particular image, generating index data that is an intermediate representation of the high-frequency residual image data; based on the low-resolution image data and an untrained set of learning parameters, generating a set of probabilities for the intermediate representation of the high-frequency residual image data; based on comparing the index data for the intermediate representation of the high-frequency residual image data with the set of probabilities for the intermediate representation of the high-frequency residual image data, updating the untrained set of learning parameters thereby generating a trained set of learning parameters. ### Claim 6 (depends on 5) The method of claim 5, further comprises: subtracting the interpolated low-resolution image data from the high-resolution image data, thereby generating the high-frequency residual image data of the particular image. ### Claim 7 (depends on 5) The method of claim 5, further comprising: determining one or more difference values between the index data for the intermediate representation of the high-frequency residual image data and the set of probabilities for the intermediate representation of the high-frequency residual image data; determining that the one or more difference values are greater than or equal to corresponding one or more predefined threshold values, or a maximum number iteration is not reached; based on determining that the one or more difference values are greater than or equal to the corresponding one or more predefined threshold values, or the maximum number iteration is not reached, selecting a next image data pair from the training dataset. ### Claim 8 (depends on 7) The method of claim 7, wherein the one or more difference values are calculated using a categorical-cross entropy. ### Claim 9 (depends on 5) The method of claim 5, wherein the untrained set of learning parameters is a second untrained set of learning parameters, the trained set of learning parameters is a second trained set of learning parameters, the high-frequency residual image data of the image is a derived high-frequency residual image data, and the method further comprising: using a first untrained set of learning parameters, generating a reproduced high-frequency residual image data of the particular image; comparing the reproduced high-frequency residual image data of the particular image with the derived high-frequency residual image data; based on comparing the reproduced high-frequency residual image data of the image with the derived high-frequency residual image data, updating the first untrained set of learning parameters thereby generating a first trained set of learning parameters; using the first trained set of learning parameters, generating the index data that is the intermediate representation of the high-frequency residual image data. ### Claim 10 (depends on 5) The method of claim 5, wherein the untrained set of learning parameters is a second untrained set of learning parameters, the trained set of learning parameters is a second trained set of learning parameters and the method further comprises: training, by a Vector-Quantized (VQ) auto-encoder, a first untrained set of learning parameters to a first trained set of learning parameters; using the first trained set of learning parameters, generating, by the VQ auto-encoder, the index data that is the intermediate representation of the high-frequency residual image data of the image. ### Claim 11 (depends on 5) The method of claim 5, further comprising: training, by a probability density estimation (PDE) logic, the untrained set of learning parameters to the trained set of learning parameters; using the trained set of learning parameters, generating, by the PDE logic, an estimated probability density of intermediate indexes. ### Claim 12 One or more non-transitory computer-readable media storing a set of instructions, wherein the set of instructions includes instructions, which, when executed by one or more hardware processors, cause: receiving input image data of a low-resolution image, having an input size, to generate output image data for a high-resolution image having an output size, the output size being greater than the input size; interpolating the input image data to match the output size, thereby generating interpolated image data for an interpolated image of the output size; determining residual image data based on the input image data, the determining the residual image data based on the input image data comprising: providing the input image data to a first convolutional neural network (CNN) trained to estimate probability density of high-resolution indexes for the input image data to generate estimated probability density of intermediate indexes; based on the estimated probability density of intermediate indexes, selecting a plurality of intermediate indexes; providing the plurality of intermediate indexes as input to a second convolutional neural network to generate the residual image data; combining the interpolated image data with the residual image data to generate the output image data for the high-resolution image of the output size. ### Claim 13 (depends on 12) The one or more non-transitory computer-readable media of claim 12, wherein the set of instructions further includes instructions, which when executed by said one or more hardware processors, cause: providing the input image data to a probability density estimation model to generate the estimated probability density of intermediate indexes; based on the estimated probability density of intermediate indexes, selecting a plurality of intermediate indexes; decoding the plurality of intermediate indexes to generate the residual image data. ### Claim 14 A system comprising one or more processors and one or more storage media storing one or more computer programs for execution by the one or more processors, the one or more computer programs configured to perform a method comprising: receiving input image data of a low-resolution image, having an input size, to generate output image data for a high-resolution image having an output size, the output size being greater than the input size; interpolating the input image data to match the output size, thereby generating interpolated image data for an interpolated image of the output size; determining residual image data based on the input image data, the determining the residual image data based on the input image data comprising: providing the input image data to a first convolutional neural network (CNN) trained to estimate probability density of high-resolution indexes for the input image data to generate estimated probability density of intermediate indexes; based on the estimated probability density of intermediate indexes, selecting a plurality of intermediate indexes; providing the plurality of intermediate indexes as input to a second convolutional neural network to generate the residual image data; combining the interpolated image data with the residual image data to generate the output image data for the high-resolution image of the output size. ### Claim 15 (depends on 14) The system of claim 14, wherein determining the residual image data based on the input image data further comprises: providing the input image data to one or more models to generate the residual image data. ### Claim 16 (depends on 14) The system of claim 14, wherein determining the residual image data based on the input image data further comprises: providing the input image data to one or more models to generate a plurality of intermediate indexes; decoding the plurality of intermediate indexes to generate the residual image data. ### Claim 17 (depends on 14) The system of claim 14, wherein determining the residual image data based on the input image data further comprises: providing the input image data to a probability density estimation model to generate the estimated probability density of intermediate indexes; based on the estimated probability density of intermediate indexes, selecting a plurality of intermediate indexes; decoding the plurality of intermediate indexes to generate the residual image data. ### Claim 18 A system comprising one or more processors and one or more storage media storing one or more computer programs for execution by the one or more processors, the one or more computer programs configured to perform a method comprising: selecting at least one image data pair from a training dataset, the at least one image data pair of the training dataset including low-resolution image data and high-resolution image data relating to a same particular image, the high-resolution image data having a desired output size; interpolating the low-resolution image data to the desired output size, thereby generating interpolated low-resolution image data of the desired output size; based on the high-resolution image data and the interpolated low-resolution image data, generating high-frequency residual image data of the particular image; based on the high-frequency residual image data of the particular image, generating index data that is an intermediate representation of the high-frequency residual image data; based on the low-resolution image data and an untrained set of learning parameters, generating a set of probabilities for the intermediate representation of the high-frequency residual image data based on comparing the index data for the intermediate representation of the high-frequency residual image data with the set of probabilities for the intermediate representation of the high-frequency residual image data, updating the untrained set of learning parameters, thereby generating a trained set of learning parameters. ### Claim 19 (depends on 18) The system of claim 18, wherein the method further comprises: subtracting the interpolated low-resolution image data from the high-resolution image data, thereby generating the high-frequency residual image data of the image. ### Claim 20 (depends on 18) The system of claim 18, wherein the method further comprises: determining one or more difference values between the index data for the intermediate representation of the high-frequency residual image data and the set of probabilities for the intermediate representation of the high-frequency residual image data; determining that the one or more difference values are greater than or equal to corresponding one or more predefined threshold values, or a maximum number iteration is not reached; based on determining that the one or more difference values are greater than or equal to the corresponding one or more predefined threshold values, or the maximum number iteration is not reached, selecting a next image data pair from the training dataset.