# US 11,907,839 B2 · Detail-preserving image editing techniques > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,839 B2 - Document ID: US11907839B2 - Status: Granted - Kind: B2 - Application number: 17/468,511 - Filed: 2021-09-07 - Published: 2022-04-21 - Granted: 2024-02-20 - Earliest claimed: 2020-10-16 - Assignees: Adobe Inc. - Applicants: Adobe Inc. - Inventors: Ratheesh Kalarot; Kevin Wampler; Jingwan Lu; Jakub Fiser; Elya Shechtman; Aliakbar Darabi; Alexandru Vasile Costin - Web page: https://patentgrep.com/patent/US11907839B2 ## CPC - G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour - G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials - G06F 18/211: Selection of the most significant subset of features - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/2163: {Partitioning the feature space} - G06F 18/40: Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0475: Generative networks - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 3/094: Adversarial learning - G06N 20/20: Ensemble learning - G06T 3/02: Affine transformations (for image registration G06T3/147; for image mosaicing G06T3/4038) - G06T 3/18: Image warping, e.g. rearranging pixels individually - G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting - G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images - G06T 3/4046: using neural networks - G06T 5/20: using local operators - G06T 5/77: Retouching; Inpainting; Scratch removal - G06T 11/00: Two-dimensional [2D] image generation - G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) - G06T 11/60: Creating or editing images; Combining images with text - G06V 10/28: Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns - G06V 10/82: using neural networks - G06V 10/98: Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns - G06V 40/168: {Feature extraction; Face representation} - G06T 2207/10024: Color image - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20221: Image fusion; Image merging - G06T 2210/22: Cropping - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Systems and methods combine an input image with an edited image generated using a generator neural network to preserve detail from the original image. A computing system provides an input image to a machine learning model to generate a latent space representation of the input image. The system provides the latent space representation to a generator neural network to generate a generated image. The system generates multiple scale representations of the input image, as well as multiple scale representations of the generated image. The system generates a first combined image based on first scale representations of the images and a first value. The system generates a second combined image based on second scale representations of the images and a second value. The system blends the first combined image with the second combined image to generate an output image. ## Claims ### Claim 1 A computer-implemented method comprising: providing, by a computing system, an input image as input to a machine learning model to generate a latent space representation of the input image; providing, by the computing system, the latent space representation of the input image as input to a trained generator neural network implemented by the computing system; generating, by the generator neural network, a generated image based on the latent space representation of the input image; generating, by the computing system, a first scale representation of the input image and a second scale representation of the input image; generating, by the computing system, a first scale representation of the generated image and a second scale representation of the generated image; generating, by the computing system, a first combined image based on the first scale representation of the input image, the first scale representation of the generated image, and a first value; generating, by the computing system, a second combined image based on the second scale representation of the input image, the second scale representation of the generated image, and a second value different from the first value; and blending, by the computing system, the first combined image with the second combined image to generate an output image. ### Claim 2 (depends on 1) The method of claim 1, further comprising: detecting, by the computing system, a first landmark in the input image; detecting, by the computing system, a second landmark in the generated image, wherein the first landmark corresponds to the second landmark; and warping, by the computing system, the input image to align the first landmark with the second landmark to produce a warped input image, wherein the warped input image is used to generate the first scale representation of the input image and the second scale representation of the input image. ### Claim 3 (depends on 1) The method of claim 1, further comprising: applying, by the computing system, edits to the latent space representation of the input image, wherein the edits are reflected in the output image. ### Claim 4 (depends on 1) The method of claim 1, further comprising outputting, by the computing system, the output image for display. ### Claim 5 (depends on 1) The method of claim 1, wherein generating the first scale representations of the images and the second scale representations of the images comprises calculating a Laplacian pyramid of the input image and a Laplacian pyramid of the generated image. ### Claim 6 (depends on 1) The method of claim 1, further comprising: generating, by the computing system, a third scale representation of the input image; generating, by the computing system, a third scale representation of the generated image; and generating, by the computing system, a third combined image based on the third scale representation of the input image, the third scale representation of the generated image, and a third value different from the first and second values, wherein generating the output image further comprises blending the first combined image and the second combined image with the third combined image. ### Claim 7 (depends on 1) The method of claim 1, wherein blending the first combined image with the second combined image to generate the output image comprises performing a Laplacian blending of the first combined image and the second combined image. ### Claim 8 (depends on 1) The method of claim 1, further comprising masking, by the computing system, a region of the generated image. ### Claim 9 A computing system comprising: a processor; a non-transitory computer-readable medium comprising instructions which, when executed by the processor, perform processing comprising: obtaining a latent space representation of an input image; providing the latent space representation of the input image as input to a trained generator neural network implemented by the computing system; generating, by the generator neural network, a generated image based on the latent space representation of the input image; generating a first scale representation of the input image and a second scale representation of the input image; generating a first scale representation of the generated image and a second scale representation of the generated image; generating a first combined image based on the first scale representation of the input image, the first scale representation of the generated image, and a first value; generating a second combined image based on the second scale representation of the input image, the second scale representation of the generated image, and a second value different from the first value; and blending the first combined image with the second combined image to generate an output image. ### Claim 10 (depends on 9) The computing system of claim 9, the processing further comprising: detecting, by the computing system, a first landmark in the input image; detecting, by the computing system, a second landmark in the generated image, wherein the first landmark corresponds to the second landmark; and warping, by the computing system, the input image to align the first landmark with the second landmark to produce a warped input image, wherein the warped input image is used to generate the first scale representation of the input image and the second scale representation of the input image. ### Claim 11 (depends on 9) The computing system of claim 9, the processing further comprising: applying, by the computing system, edits to the latent space representation of the input image, wherein the edits are reflected in the output image. ### Claim 12 (depends on 9) The computing system of claim 9, the processing further comprising: outputting the output image for display. ### Claim 13 (depends on 9) The computing system of claim 9, wherein generating the first resolution images and the second resolution images comprises calculating a Laplacian pyramid of the input image and a Laplacian pyramid of the generated image. ### Claim 14 (depends on 9) The computing system of claim 9, the processing further comprising: generating, by the computing system, a third scale representation of the input image; generating, by the computing system, a third scale representation of the generated image; and generating, by the computing system, a third combined image based on the third scale representation of the input image, the third scale representation of the generated image, and a third value different from the first and second values, wherein generating the output image further comprises blending the first combined image and the second combined image with the third combined image. ### Claim 15 (depends on 9) The computing system of claim 9, wherein blending the first combined image with the second combined image to generate the output image comprises performing a Laplacian blending of the first combined image and the second combined image. ### Claim 16 A non-transitory computer-readable medium having instructions stored thereon, the instructions executable by a processing device to perform operations comprising: providing an input image as input to a machine learning model to generate a latent space representation of the input image; providing the latent space representation of the input image as input to a trained generator neural network; generating, by the generator neural network, a generated image based on the latent space representation of the input image; and a step for producing an output image based on a first combined image of a first scale representation of the input image and a first scale representation of the generated image, and a second combined image of a second scale representation of the input image and a second scale representation of the generated image. ### Claim 17 (depends on 16) The medium of claim 16, the operations further comprising: detecting a first landmark in the input image; detecting a second landmark in the generated image, wherein the first landmark corresponds to the second landmark; and warping the input image to align the first landmark with the second landmark to produce a warped input image, wherein the warped input image is used to generate the first scale representation of the input image and the second scale representation of the input image. ### Claim 18 (depends on 16) The medium of claim 16, the operations further comprising: applying edits to the latent space representation of the input image, wherein the edits are reflected in the output image. ### Claim 19 (depends on 16) The medium of claim 16, the operations further comprising: outputting the output image for display. ### Claim 20 (depends on 16) The medium of claim 16, wherein generating the first resolution images and the second resolution images comprises calculating a Laplacian pyramid of the input image and a Laplacian pyramid of the generated image.