US 2025/0054115 A1Published application

Deep learning-based high resolution image inpainting

AssigneeAdobe Inc.

InventorsZhe LIN, Yuqian ZHOU, Sohrab AMIRGHODSI, Qing LIU, Elya SHECHTMAN, Connelly BARNES, Haitian ZHENG

Life of this patent

Estimating the term…

Loading patent…

Abstract

Various disclosed embodiments are directed to resizing, via down-sampling and up-sampling, a high-resolution input image in order to meet machine learning model low-resolution processing requirements, while also producing a high-resolution output image for image inpainting via a machine learning model. Some embodiments use a refinement model to refine the low-resolution inpainting result from the machine learning model such that there will be clear content with high resolution both inside and outside of the mask region in the output. Some embodiments employ new model architecture for the machine learning model that produces the inpainting result—an advanced Cascaded Modulated Generative Adversarial Network (CM-GAN) that includes Fast Fourier Convolution (FCC) layers at the skip connections between the encoder and decoder.

Claims20

Dependent claims

A system comprising:

at least one computer processor;
and
one or more computer storage media storing computer-useable instructions that, when used by the at least one computer processor, cause the at least one computer processor to perform operations comprising:
receiving a first image at a first resolution, the first image including a mask that is indicative of a hole associated with image inpainting;
generating a second image by downsampling the first image to a second resolution that is lower than the first resolution;
in response to the generating of the second image, providing the second image as input to a first machine learning model;
based on the providing, receiving a third image at the second resolution, the third image being included in an output of the first machine learning model, the third image being identical to the second image except that the mask is replaced by a first set of pixels, the replacement of the mask by the first set of pixels being indicative of image inpainting;
and
in response to the receiving of the third image, generating a fourth image by upsampling the third image, the upsampling of the third image includes generating one or more portions of the fourth image at the first resolution.

6 more elements

A computer-implemented method comprising:

receiving a first image at a first resolution, the first image including a mask that is indicative of a hole associated with image inpainting;
generating a second image by downsampling the first image to a second resolution that is lower than the first resolution;
in response to the generating of the second image, inpainting the second image by replacing the mask with a first set of pixels;
and
in response to the inpainting of the second image, generating a third image by upsampling the inpainted second image, the upsampling includes generating one or more portions outside of the first set of pixels back to the first resolution.

2 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/232,212Filed 9 Aug 2023
Published: 13 Feb 2025
Term ends: Not estimatedThe source records don’t support an estimate.

Classification12 codes

G06T 5/77
Image enhancement or restoration: retouching; Inpainting; Scratch removal
G06T 3/4046
Scaling of whole images or parts thereof, e.g. expanding or contracting: using neural networks
G06T 3/4053
Scaling of whole images or parts thereof, e.g. expanding or contracting: based on super-resolution, i.e. the output image resolution being higher…
G06T 5/60
Image enhancement or restoration: using machine learning, e.g. neural networks