# US 2026/0089303 A1 · Photorealistic 4d scene generation using video diffusion models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0089303 A1
- Document ID: US20260089303A1
- Status: Published application
- Kind: A1
- Application number: 18/893,343
- Filed: 2024-09-23
- Published: 2026-03-26
- Earliest claimed: 2024-09-23
- Assignees: Snap Inc.
- Applicants: Snap Inc.
- Inventors: Hsin-Ying Lee; Willi Menapace; Aliaksandr Siarohin; Sergey Tulyakov; Chaoyang Wang; Heng Yu; Peiye Zhuang
- Web page: https://patentgrep.com/patent/US20260089303A1

## CPC

- H04N 13/279: the virtual viewpoint locations being selected by the viewers or determined by tracking
- G06T 15/20: Perspective computation
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
- H04N 21/816: {involving special video data, e.g 3D video}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method for generating photorealistic 4D scenes from text inputs is disclosed. The method utilizes a text-to-video diffusion model to generate a reference video and a freeze-time video. A canonical 3D representation is reconstructed using deformable 3D Gaussian Splats (D-3DGS) based on the freeze-time video. Temporal deformations are learned to capture dynamic interactions in the reference video. The method employs a novel Score Distillation Sampling strategy combining multi-view and temporal aspects to enhance consistency and robustness. The resulting 4D scenes feature multiple objects interacting with detailed background environments, viewable from different angles and times. The method enables flexible camera control and integration with augmented and virtual reality applications. Some examples include features such as image-to-4D generation.

## Claims

### Claim 1

A system comprising:
at least one processor; and
a memory storing instructions that, when executed by the at least one processor, cause the system to perform operations comprising:
obtaining a 4D scene model comprising:
a 3D representation of a scene, the scene comprising at least one action paused at a point in time; and
a plurality of temporal deformations corresponding to transformations of the 3D representation to show the scene at different points in time during the action;
the 3D representation and temporal deformations being generated based on a reference video showing the action occurring in the scene; and
applying a point of view transformation and at least one temporal deformation, selected from the plurality of temporal deformations, to the 3D representation to generate one or more views of the scene.

### Claim 2 (depends on 1)

The system of claim 1, wherein:
obtaining the 4D scene model comprises:
obtaining a reference video comprising a first sequence of frames showing the scene comprising at least one action occurring over a time period corresponding to the first sequence of frames;
processing the reference video to generate a freeze time video, the freeze time video comprising a second sequence of frames showing the scene of the first sequence of frames from a plurality of points of view, the at least one action of the scene being shown in the second sequence of frames paused at a time corresponding to a frame of the first sequence of frames;
processing the freeze time video to generate the 3D representation of the scene; and
processing the reference video and the 3D representation to generate the plurality of temporal deformations.

### Claim 3 (depends on 2)

The system of claim 2, wherein:
obtaining the reference video comprises generating the reference video based on an input using a diffusion model trained to generate videos.

### Claim 4 (depends on 3)

The system of claim 3, wherein:
the input comprises text; and
the diffusion model comprises a text-to-video diffusion model.

### Claim 5 (depends on 3)

The system of claim 3, wherein:
the input comprises an image; and
the diffusion model comprises an image-to-video diffusion model.

### Claim 6 (depends on 2)

The system of claim 2, wherein:
the processing of the reference video and the 3D representation to generate the plurality of temporal deformations comprises:
learning the plurality of temporal deformations by training the 4D scene model to reconstruct the first sequence of frames based on the 3D representation.

### Claim 7 (depends on 6)

The system of claim 6, wherein:
the processing of the reference video and the 3D representation to generate the plurality of temporal deformations further comprises:
learning a plurality of per-frame deformations by training the 4D scene model to reconstruct the second sequence of frames based on the 3D representation; and
updating the 3D representation while learning the plurality of temporal deformations and the plurality of per-frame deformations.

### Claim 8 (depends on 6)

The system of claim 6, wherein:
the processing of the reference video, and the 3D representation to generate the plurality of temporal deformations further comprises:
generating one or more multi-view SDS videos showing the scene from multiple points of view;
generating one or more temporal SDS videos showing the scene at multiple times during the action;
adding noise to the one or more multi-view SDS videos and the one or more temporal SDS videos to generate noised videos;
for each noised video, processing the noised video using a video diffusion model to generate a corresponding denoised video;
generating an SDS loss based on a difference between the noised video and the corresponding denoised video; and
further training the 4D scene model using the SDS loss.

### Claim 9 (depends on 1)

The system of claim 1, wherein:
the 3D representation comprises a 3D Gaussian splat representation.

### Claim 10 (depends on 1)

The system of claim 1, wherein:
the generating of the one or more views of the scene comprises:
receiving pose data from an XR device;
generating a point of view transformation based on the pose data;
applying the point of view transformation to the 3D representation to generate at least one view of the scene from at least one viewpoint corresponding to the pose data; and
displaying the at least one view on the XR device in real-time.

### Claim 11 (depends on 1)

The system of claim 1, wherein:
the one or more views of the scene are photorealistic views.

### Claim 12 (depends on 1)

The system of claim 1, wherein:
the one or more views of the scene include views of an environment where the action occurs.

### Claim 13

A method comprising:
obtaining a 4D scene model comprising:
a 3D representation of a scene, the scene comprising at least one action paused at a point in time; and
a plurality of temporal deformations corresponding to transformations of the 3D representation to show the scene at different points in time during the action;
the 3D representation and temporal deformations being generated based on a reference video showing the action occurring in the scene; and
applying a point of view transformation and at least one temporal deformation, selected from the plurality of temporal deformations, to the 3D representation to generate one or more views of the scene.

### Claim 14 (depends on 13)

The method of claim 13, wherein:
obtaining the 4D scene model comprises:
obtaining a reference video comprising a first sequence of frames showing the scene comprising at least one action occurring over a time period corresponding to the first sequence of frames;
processing the reference video to generate a freeze time video, the freeze time video comprising a second sequence of frames showing the scene of the first sequence of frames from a plurality of points of view, the at least one action of the scene being shown in the second sequence of frames paused at a time corresponding to a frame of the first sequence of frames;
processing the freeze time video to generate the 3D representation of the scene; and
processing the reference video and the 3D representation to generate the plurality of temporal deformations.

### Claim 15 (depends on 14)

The method of claim 14, wherein:
obtaining the reference video comprises generating the reference video based on an input using a transformer model trained to generate videos.

### Claim 16 (depends on 14)

The method of claim 14, wherein:
the processing of the reference video and the 3D representation to generate the plurality of temporal deformations comprises:
learning the plurality of temporal deformations by training the 4D scene model to reconstruct the first sequence of frames based on the 3D representation.

### Claim 17 (depends on 16)

The method of claim 16, wherein:
the processing of the reference video and the 3D representation to generate the plurality of temporal deformations further comprises:
generating one or more multi-view SDS videos showing the scene from multiple points of view;
generating one or more temporal SDS videos showing the scene at multiple times during the action;
adding noise to the one or more multi-view SDS videos and the one or more temporal SDS videos to generate noised videos;
for each noised video, processing the noised video using a video diffusion model to generate a corresponding denoised video;
generating an SDS loss based on a difference between the noised video and the corresponding denoised video; and
further training the 4D scene model using the SDS loss.

### Claim 18 (depends on 13)

The method of claim 13, wherein:
the 3D representation comprises a 3D Gaussian splat representation.

### Claim 19 (depends on 13)

The method of claim 13, wherein:
the generating of the one or more views of the scene comprises:
receiving pose data from an XR device;
generating a point of view transformation based on the pose data;
applying the point of view transformation to the 3D representation to generate at least one view of the scene from at least one viewpoint corresponding to the pose data; and
displaying the at least one view on the XR device in real-time.

### Claim 20

A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that, when executed by at least one processor of a system, cause the system to perform operations comprising:
obtaining a 4D scene model comprising:
a 3D representation of a scene, the scene comprising at least one action paused at a point in time; and
a plurality of temporal deformations corresponding to transformations of the 3D representation to show the scene at different points in time during the action; the 3D representation and temporal deformations being generated based on a reference video showing the action occurring in the scene; and
applying a point of view transformation and at least one temporal deformation, selected from the plurality of temporal deformations, to the 3D representation to generate one or more views of the scene.
