# US 2025/0378633 A1 · Relightable 3D Reconstruction and View Synthesis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0378633 A1
- Document ID: US20250378633A1
- Status: Published application
- Kind: A1
- Application number: 19/230,885
- Filed: 2025-06-06
- Published: 2025-12-11
- Earliest claimed: 2024-06-06
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Philipp Henzler; Ricardo Martin-Brualla; Xiaoming Zhao; Pratul Preeti Srinivasan; Dor Verbin; Keunhong Park
- Web page: https://patentgrep.com/patent/US20250378633A1

## CPC

- G06T 15/506: {Illumination models}
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 5/70: Denoising; Smoothing
- G06T 15/20: Perspective computation
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Provided are systems and methods for relightable view synthesis that can process a set of source images captured under unknown lighting conditions to produce 3D reconstructions under novel target lighting and from novel viewpoints or poses. Initially, an example method includes obtaining source images and target lighting data, followed by generating radiance data using a source neural scene representation and a rendering engine. A machine-learned relighting diffusion model can then be employed to process the source images and radiance data to generate re-lit images. These images are subsequently used to train a latent neural radiance field model, which, upon querying following training, can generate synthetic images from novel poses under the target lighting. The proposed technology can be beneficial for applications in virtual reality, filmmaking, game development, and other settings, offering a robust alternative to traditional inverse rendering methods by leveraging advanced machine learning techniques to handle complex lighting scenarios.

## Claims

### Claim 1

A computer-implemented method for performing relightable view synthesis, the method comprising:
obtaining, by a computing system comprising one or more computing devices, (i) a plurality of source images that depict a scene with a source lighting and (ii) target lighting data that describes a target lighting for the scene, the target lighting being different from the source lighting;
generating, by the computing system and based on the plurality of source images, radiance data that represents radiance characteristics of the scene under the target lighting;
respectively processing, by the computing system, the plurality of source images and the radiance data with a machine-learned relighting diffusion model to respectively generate a plurality of re-lit images that depict the scene with the target lighting;
training, by the computing system, a latent neural radiance field model using the plurality of re-lit images; and
after training latent neural radiance field model, querying, by the computing system, the latent neural radiance field model to generate a synthetic image that depicts the scene with the target lighting from a novel pose.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein generating, by the computing system and based on the plurality of source images, the radiance data that represents radiance characteristics of the scene under the target lighting comprises:
training, by the computing system, a source neural scene representation using the plurality of source images;
after training the source neural scene representation, generating, by the computing system, scene surface information as an output of the source neural radiance field; and
processing, by the computing system, the scene surface information and the target lighting data with a rendering engine to generate the radiance data.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein the radiance data comprises a plurality of renderings of a surface geometry of the scene under the target lighting, wherein in the plurality of renderings the surface geometry of the scene respectively has a plurality of different material characteristics.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, further comprising, prior to processing, by the computing system, the plurality of source images and the radiance data with the machine-learned relighting diffusion model:
obtaining, by the computing system, a relighting training dataset comprising a plurality of training examples, each training example comprising (i) a training source image that depicts a training scene under a training source lighting, (ii) training radiance cues that describe radiance characteristics of the training scene under a training target lighting, and (iii) a training re-lit image that depicts the training scene under the training target lighting; and
training, by the computing system, the machine-learned relighting diffusion model to generate the training re-lit image from input noise conditioned on the training source image and the training radiance cues.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein obtaining, by the computing system, the relighting training dataset comprises generating the relighting training dataset using a set of three-dimensional rendering assets and a rendering engine.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein training, by the computing system, the latent neural radiance field model using the plurality of re-lit images comprises:
respectively initializing, by the computing system, a plurality of latent variable values for the plurality of re-lit images; and
jointly optimizing, by the computing system and using the plurality of re-lit images, (i) parameter values of the latent neural radiance field model and (ii) the plurality of latent variable values for the plurality of re-lit images.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein querying, by the computing system, the latent neural radiance field model to generate the synthetic image comprises querying, by the computing system, the latent neural radiance field model with (i) pose data describing the novel pose and (ii) a latent variable query value.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the source lighting comprises an unknown lighting.

### Claim 9

A computing system comprising one or more processors and one or more non-transitory computer-readable media that collectively store instructions that when executed by the one or more processors cause the computing system to perform operations, the operations comprising:
obtaining, by the computing system comprising, (i) a plurality of source images that depict a scene with a source lighting and (ii) target lighting data that describes a target lighting for the scene, the target lighting being different from the source lighting;
generating, by the computing system and based on the plurality of source images, radiance data that represents radiance characteristics of the scene under the target lighting;
respectively processing, by the computing system, the plurality of source images and the radiance data with a machine-learned relighting diffusion model to respectively generate a plurality of re-lit images that depict the scene with the target lighting;
training, by the computing system, a latent neural radiance field model using the plurality of re-lit images; and
after training latent neural radiance field model, querying, by the computing system, the latent neural radiance field model to generate a synthetic image that depicts the scene with the target lighting from a novel pose.

### Claim 10 (depends on 9)

The computing system of claim 9, wherein generating, by the computing system and based on the plurality of source images, the radiance data that represents radiance characteristics of the scene under the target lighting comprises:
training, by the computing system, a source neural scene representation using the plurality of source images;
after training the source neural scene representation, generating, by the computing system, scene surface information as an output of the source neural radiance field; and
processing, by the computing system, the scene surface information and the target lighting data with a rendering engine to generate the radiance data.

### Claim 11 (depends on 10)

The computing system of claim 10, wherein the radiance data comprises a plurality of renderings of a surface geometry of the scene under the target lighting, wherein in the plurality of renderings the surface geometry of the scene respectively has a plurality of different material characteristics.

### Claim 12 (depends on 9)

The computing system of claim 9, wherein the operations further comprise, prior to processing, by the computing system, the plurality of source images and the radiance data with the machine-learned relighting diffusion model:
obtaining, by the computing system, a relighting training dataset comprising a plurality of training examples, each training example comprising (i) a training source image that depicts a training scene under a training source lighting, (ii) training radiance cues that describe radiance characteristics of the training scene under a training target lighting, and (iii) a training re-lit image that depicts the training scene under the training target lighting; and
training, by the computing system, the machine-learned relighting diffusion model to generate the training re-lit image from input noise conditioned on the training source image and the training radiance cues.

### Claim 13 (depends on 12)

The computing system of claim 12, wherein obtaining, by the computing system, the relighting training dataset comprises generating the relighting training dataset using a set of three-dimensional rendering assets and a rendering engine.

### Claim 14 (depends on 9)

The computing system of claim 9, wherein training, by the computing system, the latent neural radiance field model using the plurality of re-lit images comprises:
respectively initializing, by the computing system, a plurality of latent variable values for the plurality of re-lit images; and
jointly optimizing, by the computing system and using the plurality of re-lit images, (i) parameter values of the latent neural radiance field model and (ii) the plurality of latent variable values for the plurality of re-lit images.

### Claim 15 (depends on 14)

The computing system of claim 14, wherein querying, by the computing system, the latent neural radiance field model to generate the synthetic image comprises querying, by the computing system, the latent neural radiance field model with (i) pose data describing the novel pose and (ii) a latent variable query value.

### Claim 16 (depends on 9)

The computing system of claim 9, wherein the source lighting comprises an unknown lighting.

### Claim 17

One or more non-transitory computer-readable media that store a latent neural radiance field model configured to generate a synthetic image that depicts the scene with the target lighting from a novel pose, wherein the latent neural radiance field model has previously been trained by performance of training operations, the training operations comprising:
obtaining, by a computing system comprising one or more computing devices, (i) a plurality of source images that depict a scene with a source lighting and (ii) target lighting data that describes a target lighting for the scene, the target lighting being different from the source lighting;
generating, by the computing system and based on the plurality of source images, radiance data that represents radiance characteristics of the scene under the target lighting;
respectively processing, by the computing system, the plurality of source images and the radiance data with a machine-learned relighting diffusion model to respectively generate a plurality of re-lit images that depict the scene with the target lighting; and
training, by the computing system, the latent neural radiance field model using the plurality of re-lit images.

### Claim 18 (depends on 17)

The one or more non-transitory computer-readable media of claim 17, wherein generating, by the computing system and based on the plurality of source images, the radiance data that represents radiance characteristics of the scene under the target lighting comprises:
training, by the computing system, a source neural scene representation using the plurality of source images;
after training the source neural scene representation, generating, by the computing system, scene surface information as an output of the source neural radiance field; and
processing, by the computing system, the scene surface information and the target lighting data with a rendering engine to generate the radiance data.

### Claim 19 (depends on 17)

The one or more non-transitory computer-readable media of claim 17, wherein the training operations further comprise, prior to processing, by the computing system, the plurality of source images and the radiance data with the machine-learned relighting diffusion model:
obtaining, by the computing system, a relighting training dataset comprising a plurality of training examples, each training example comprising (i) a training source image that depicts a training scene under a training source lighting, (ii) training radiance cues that describe radiance characteristics of the training scene under a training target lighting, and (iii) a training re-lit image that depicts the training scene under the training target lighting; and
training, by the computing system, the machine-learned relighting diffusion model to generate the training re-lit image from input noise conditioned on the training source image and the training radiance cues.

### Claim 20 (depends on 17)

The one or more non-transitory computer-readable media of claim 17, wherein training, by the computing system, the latent neural radiance field model using the plurality of re-lit images comprises:
respectively initializing, by the computing system, a plurality of latent variable values for the plurality of re-lit images; and
jointly optimizing, by the computing system and using the plurality of re-lit images, (i) parameter values of the latent neural radiance field model and (ii) the plurality of latent variable values for the plurality of re-lit images.
