# US 12,008,710 B2 · Generating light-source-specific parameters for digital images using a neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,710 B2
- Document ID: US12008710B2
- Status: Granted
- Kind: B2
- Application number: 18/062,460
- Filed: 2022-12-06
- Published: 2023-03-30
- Granted: 2024-06-11
- Earliest claimed: 2019-09-03
- Assignees: Adobe Inc.; Universite Laval
- Applicants: Adobe Inc.; Université Laval
- Inventors: Kalyan Sunkavalli; Yannick Hold-Geoffroy; Christian Gagne; Marc-Andre Gardner; Jean-Francois Lalonde
- Web page: https://patentgrep.com/patent/US12008710B2

## CPC

- G06T 15/506: {Illumination models}
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06T 7/50: Depth or shape recovery
- G06T 7/60: Analysis of geometric attributes
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/90: Determination of colour characteristics
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2219/2012: Colour editing, changing, or manipulating; Use of colour codes
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

This disclosure relates to methods, non-transitory computer readable media, and systems that can render a virtual object in a digital image by using a source-specific-lighting-estimation-neural network to generate three-dimensional (“3D”) lighting parameters specific to a light source illuminating the digital image. To generate such source-specific-lighting parameters, for instance, the disclosed systems utilize a compact source-specific-lighting-estimation-neural network comprising both common network layers and network layers specific to different lighting parameters. In some embodiments, the disclosed systems further train such a source-specific-lighting-estimation-neural network to accurately estimate spatially varying lighting in a digital image based on comparisons of predicted environment maps from a differentiable-projection layer with ground-truth-environment maps.

## Claims

### Claim 1

A non-transitory computer-readable medium storing computer-readable instructions which, when executed by at least one processing device, cause the at least one processing device to perform operations comprising:
extracting a latent feature vector from a digital image utilizing a first subset of common network layers of a source-specific-lighting-estimation-neural network;
extracting a common feature vector from the latent feature vector utilizing a second subset of common network layers of the source-specific-lighting-estimation-neural network; and
generating three-dimensional (“3D”) source-specific-lighting parameters based on the common feature vector utilizing parametric-specific-network layers of the source-specific-lighting-estimation-neural network by:
generating 3D-source-specific-distance parameters estimating one or more distances of one or more light sources from a reference point based on the latent feature vector utilizing distance-parametric-specific-network layers of the parametric-specific-network layers; and
generating source-specific-lighting parameters based on the common feature vector utilizing additional parametric-specific-network layers of the parametric-specific-network layers.

### Claim 2 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein generating the source-specific-lighting parameters comprises applying different sets of the additional parametric-specific-network layers to generate:
3D-source-specific-direction parameters estimating one or more directions of the one or more light sources with respect to the reference point;
source-specific-size parameters estimating one or more sizes of the one or more light sources; and
source-specific-color parameters estimating one or more colors of the one or more light sources.

### Claim 3 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein generating the 3D-source-specific-distance parameters comprises:
generating a first set of 3D-source-specific-distance parameters corresponding to a first predicted light source illuminating the digital image; and
generating a second set of 3D-source-specific-distance parameters corresponding to a second predicted light source illuminating the digital image.

### Claim 4 (depends on 2)

The non-transitory computer-readable medium of claim 2, wherein the first subset of common network layers comprises an encoder from a densely connected convolutional network, the second subset of common network layers comprises fully connected layers, and each set of the parametric-specific-network layers comprise a parametric-specific decoder.

### Claim 5 (depends on 4)

The non-transitory computer-readable medium of claim 4, wherein the operations further comprise generating the 3D-source-specific-distance parameters based on the latent feature vector and the 3D-source-specific-direction parameters utilizing distance-parametric-specific-network layers from the parametric-specific-network layers.

### Claim 6 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the operations further comprise providing, for display on a computing device, a graphical user interface comprising:
a first set of lighting parameter controls for a first set of 3D-source-specific-lighting parameters corresponding to a first predicted light source illuminating a modified digital image; and
a second set of lighting parameter controls for a second set of 3D-source-specific-lighting parameters corresponding to a second predicted light source illuminating the modified digital image.

### Claim 7 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the operations further comprise:
identifying a position-adjustment request to move a virtual object from a designated position within the digital image to a new designated position within the digital image;
adjusting a projection of the 3D-source-specific-lighting parameters for the new designated position within the digital image; and
based on the position-adjustment request, rendering an adjusted digital image comprising the virtual object at the new designated position illuminated according to the adjusted projection of the 3D-source-specific-lighting parameters.

### Claim 8 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein:
generating the 3D source-specific-lighting parameters comprises inferring the 3D-source-specific-lighting parameters directly from the common feature vector; and
the source-specific-lighting-estimation-neural network comprises less than ten million network parameters.

### Claim 9

A system comprising:
one or more memory devices storing a digital image and a source-specific-lighting-estimation-neural network; and
one or more server devices that cause the system to:
extract a latent feature vector from the digital image utilizing a first subset of common network layers of the source-specific-lighting-estimation-neural network;
extract a common feature vector from the latent feature vector utilizing a second subset of common network layers of the source-specific-lighting-estimation-neural network; and
generate three-dimensional (“3D”) source-specific-lighting parameters based on the common feature vector utilizing parametric-specific-network layers of the source-specific-lighting-estimation-neural network by:
generating 3D-source-specific-distance parameters estimating one or more distances of one or more light sources from a reference point based on the latent feature vector utilizing distance-parametric-specific-network layers of the parametric-specific-network layers; and
generating source-specific-lighting parameters based on the common feature vector utilizing additional parametric-specific-network layers of the parametric-specific-network layers.

### Claim 10 (depends on 9)

The system of claim 9, wherein generating the source-specific-lighting parameters comprises applying different sets of the additional parametric-specific-network layers to generate:
3D-source-specific-direction parameters estimating one or more directions of the one or more light sources with respect to the reference point;
source-specific-size parameters estimating one or more sizes of the one or more light sources; and
source-specific-color parameters estimating one or more colors of the one or more light sources.

### Claim 11 (depends on 9)

The system of claim 9, wherein generating the 3D-source-specific-distance parameters comprises:
generating a first set of 3D-source-specific-distance parameters corresponding to a first predicted light source illuminating the digital image; and
generating a second set of 3D-source-specific-distance parameters corresponding to a second predicted light source illuminating the digital image.

### Claim 12 (depends on 9)

The system of claim 9, wherein the first subset of common network layers comprises an encoder from a densely connected convolutional network, the second subset of common network layers comprises fully connected layers, and each set of the parametric-specific-network layers comprise a parametric-specific decoder.

### Claim 13 (depends on 9)

The system of claim 9, wherein the one or more server devices further cause the system to:
identify a request to render a virtual object at a designated position within the digital image; and
based on the request, render a modified digital image comprising the virtual object at the designated position illuminated according to the 3D-source-specific-lighting parameters.

### Claim 14 (depends on 13)

The system of claim 13, wherein the one or more server devices further cause the system to:
provide, for display on a computing device, a graphical user interface comprising a set of lighting parameter controls for a set of 3D-source-specific-lighting parameters corresponding to a predicted light source illuminating the modified digital image; and
based on detecting a user interaction with a lighting parameter control from among the set of lighting parameter controls, adjust one or more 3D-source-specific-lighting parameters from the set of 3D-source-specific-lighting parameters corresponding to the predicted light source.

### Claim 15

A method comprising:
extracting a latent feature vector from a digital image utilizing a first subset of common network layers of a source-specific-lighting-estimation-neural network;
extracting a common feature vector from the latent feature vector utilizing a second subset of common network layers of the source-specific-lighting-estimation-neural network; and
generating three-dimensional (“3D”) source-specific-lighting parameters based on the common feature vector utilizing parametric-specific-network layers of the source-specific-lighting-estimation-neural network by:
generating 3D-source-specific-distance parameters estimating one or more distances of one or more light sources from a reference point based on the latent feature vector utilizing distance-parametric-specific-network layers of the parametric-specific-network layers; and
generating source-specific-lighting parameters based on the common feature vector utilizing additional parametric-specific-network layers of the parametric-specific-network layers.

### Claim 16 (depends on 15)

The method of claim 15, further comprising training the source-specific-lighting-estimation-neural network in a first training stage by:
generating 3D-source-specific-predicted-lighting parameters based on a common-feature-training vector extracted from a digital training image utilizing the source-specific-lighting-estimation-neural network;
applying a differentiable-projection layer to the 3D-source-specific-predicted-lighting parameters to project a predicted environment map corresponding to the digital training image;
comparing the predicted environment map to a ground-truth-environment map corresponding to the digital training image to determine an environment-map loss; and
modifying internal parameters of the source-specific-lighting-estimation-neural network based on the environment-map loss.

### Claim 17 (depends on 16)

The method of claim 16, further comprising training the source-specific-lighting-estimation-neural network in a second training stage by:
generating subsequent 3D-source-specific-predicted-lighting parameters based on a subsequent common-feature-training vector extracted from a subsequent digital training image utilizing the source-specific-lighting-estimation-neural network;
comparing the subsequent 3D-source-specific-predicted-lighting parameters to ground-truth-source-specific-lighting parameters corresponding to the subsequent digital training image to determine lighting parameter losses; and
modifying internal parameters of the parametric-specific-network layers based on the lighting parameter losses while maintaining internal parameters of the first and second subsets of common network layers.

### Claim 18 (depends on 17)

The method of claim 17, wherein training the source-specific-lighting-estimation-neural network comprises:
adding the distance-parametric-specific-network layers to the parametric-specific-network layers for the second training stage; and
generating 3D-source-specific-predicted-distance parameters as part of the subsequent 3D-source-specific-predicted-lighting parameters, the 3D-source-specific-predicted-distance parameters estimating one or more distances of one or more predicted light sources from the reference point utilizing the distance-parametric-specific-network layers.

### Claim 19 (depends on 17)

The method of claim 17, wherein comparing the subsequent 3D-source-specific-predicted-lighting parameters to the ground-truth-source-specific-lighting parameters to determine the lighting parameter losses comprises:
comparing subsequent 3D-source-specific-predicted-distance parameters for a predicted light source to ground-truth-source-specific-distance parameters to determine a distance-parameter loss;
comparing subsequent 3D-source-specific-predicted-size parameters for the predicted light source to ground-truth-source-specific-size parameters to determine a size-parameter loss; and
comparing subsequent 3D-source-specific-predicted-color parameters for the predicted light source to ground-truth-source-specific-color parameters to determine a color-parameter loss.

### Claim 20 (depends on 15)

The method of claim 15, wherein generating the source-specific-lighting parameters comprises applying different sets of the additional parametric-specific-network layers to generate:
3D-source-specific-direction parameters estimating one or more directions of the one or more light sources with respect to the reference point;
source-specific-size parameters estimating one or more sizes of the one or more light sources; and
source-specific-color parameters estimating one or more colors of the one or more light sources.