# US 2022/0036534 A1 · Facial reconstruction network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036534 A1
- Document ID: US20220036534A1
- Status: Published application
- Kind: A1
- Application number: 16/944,452
- Filed: 2020-07-31
- Published: 2022-02-03
- Earliest claimed: 2020-07-31
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: SAIKAT CHAKRABARTY; SUNIL KUMAR
- Web page: https://patentgrep.com/patent/US20220036534A1

## CPC

- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06V 10/82: using neural networks
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 10/77: Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20104: Interactive definition of region of interest [ROI]
- G06T 2207/20212: Image combination
- G06T 2207/30201: Face
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods for digital image processing are described. Embodiments of the systems and methods includes identifying a digital image that includes an obscured region; generating a predicted image using a neural network comprising an encoder, a feature transfer network, and a decoder, wherein the feature transfer network is trained to predict features corresponding to the obscured region based on an output of the decoder; and combining the digital image and the predicted image to produce a reconstructed version of the digital image including details corresponding to the obscured region that are not present in the digital image.

## Claims

### Claim 1

A method for image processing, the method comprising:
identifying a digital image that includes an obscured region;
generating a predicted image using a neural network comprising an encoder, a feature transfer network, and a decoder, wherein the feature transfer network is trained to predict features corresponding to the obscured region based on an output of the decoder; and
combining the digital image and the predicted image to produce a reconstructed version of the digital image including details corresponding to the obscured region that are not present in the digital image.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
receiving an input from a user identifying a region of the digital image; and
obscuring the region based on an obscuring technique, wherein the feature transfer network is trained based on a training set including images having regions obscured using the obscuring technique.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
generating an image encoding based on the digital image using the encoder;
generating a reconstructed encoding based on the image encoding using the feature transfer network, wherein the reconstructed image includes the predicted features; and
generating the predicted image based on the reconstructed encoding using the decoder.

### Claim 4 (depends on 1)

The method of claim 1, wherein:
the encoder, the feature transfer network, and the decoder are each trained independently.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
generating an intermediate latent vector based on the digital image using the encoder; and
tuning the intermediate latent vector using an optimizer, wherein an output of the encoder is based on the tuned intermediate latent vector.

### Claim 6 (depends on 5)

The method of claim 5, wherein:
the intermediate latent vector is tuned using the optimizer based on a mean square error (MSE) loss between the digital image and the output of the encoder.

### Claim 7 (depends on 6)

The method of claim 6, wherein:
the MSE loss is based on a grayscale mask representing pixels of the digital image.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
generating an image encoding based on the digital image using the encoder;
generating a reconstructed image based on the image encoding using the decoder; and
tuning the encoder based on a comparison of the image encoding and the reconstructed image, wherein the encoder is tuned using a perceptual loss function that takes the image encoding and the reconstructed image as inputs.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
receiving a quality input from a user; and
updating learning parameters based on the quality input, wherein the encoder is tuned based on the learning parameters.

### Claim 10

A method of training a network for image processing, the method comprising:
training a decoder to generate images based on input vectors;
training an encoder to generate feature vectors based on input images, wherein the encoder is trained based at least in part on the trained decoder; and
training a feature transfer network to generate reconstructed feature vectors based on the feature vectors, wherein the reconstructed feature vectors comprise details corresponding to obscured regions of the feature vectors, and wherein the feature transfer network is trained based at least in part on the trained encoder.

### Claim 11 (depends on 10)

The method of claim 10, wherein:
the decoder is trained along with a discriminative network according to a generative adversarial network (GAN) framework.

### Claim 12 (depends on 10)

The method of claim 10, further comprising:
generating the images based on the input vectors using the decoder;
generating the feature vectors based on the generated images; and
comparing the feature vectors and the input vectors.

### Claim 13 (depends on 10)

The method of claim 10, further comprising:
generating the feature vectors based on a set of input images using the encoder;
obscuring regions of the input images to create a set of obscured input images;
generating obscured feature vectors based on the obscured input images using the encoder;
generating the reconstructed feature vectors based on the obscured feature vectors using the feature transfer network; and
comparing the feature vectors to the reconstructed feature vectors.

### Claim 14 (depends on 10)

The method of claim 10, further comprising:
identifying an additional image comprising an obscured region; and
refining an intermediate latent vector of the encoder using an optimizer based on a mean square error (MSE) loss between the additional image and an output of the encoder and a grayscale image mask based on the obscured region.

### Claim 15

An apparatus for image processing, comprising:
an encoder configured to generate feature vectors based on input images that include obscured regions;
a feature transfer network configured to generate reconstructed feature vectors based on the feature vectors, wherein the reconstructed feature vectors include reconstructed features corresponding to the obscured regions of the input images; and
a decoder configured to generate reconstructed images based on the reconstructed feature vectors, wherein the reconstructed images include details corresponding to the obscured regions of the input images based on the reconstructed features.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein:
the encoder comprises a deep convolutional neural network (CNN).

### Claim 17 (depends on 15)

The apparatus of claim 15, wherein:
the feature transfer network comprises an input layer, a hidden layer, and an output layer, wherein the hidden layer comprises twice as many nodes as the input layer.

### Claim 18 (depends on 15)

The apparatus of claim 15, wherein:
the decoder comprises a generative adversarial network (GAN).

### Claim 19 (depends on 18)

The apparatus of claim 18, wherein:
the decoder comprises a mapping network and a synthesis network, and the encoder is trained by comparing outputs of the mapping network to outputs of the encoder.

### Claim 20 (depends on 15)

The apparatus of claim 15, further comprising:
an optimizer configured to refine an intermediate latent vector of the encoder.
