US 12,701,209 B2 · Photorealistic 4D scene generation using video diffusion models

Abstract

A method for generating photorealistic 4D scenes from text inputs is disclosed. The method utilizes a text-to-video diffusion model to generate a reference video and a freeze-time video. A canonical 3D representation is reconstructed using deformable 3D Gaussian Splats (D-3DGS) based on the freeze-time video. Temporal deformations are learned to capture dynamic interactions in the reference video. The method employs a novel Score Distillation Sampling strategy combining multi-view and temporal aspects to enhance consistency and robustness. The resulting 4D scenes feature multiple objects interacting with detailed background environments, viewable from different angles and times. The method enables flexible camera control and integration with augmented and virtual reality applications. Some examples include features such as image-to-4D generation.

Claims20

Dependent claims

A system comprising:

at least one processor;
and
a memory storing instructions that, when executed by the at least one processor, cause the system to perform operations comprising:
obtaining a 4D scene model comprising:
a 3D representation of a scene, the scene comprising at least one action paused at a point in time;
and
a plurality of temporal deformations corresponding to transformations of the 3D representation to show the scene at different points in time during the action;
the 3D representation and temporal deformations being generated based on a reference video showing the action occurring in the scene;
and
applying a point of view transformation and at least one temporal deformation, selected from the plurality of temporal deformations, to the 3D representation to generate one or more views of the scene.

7 more elements

A method comprising:

obtaining a 4D scene model comprising:
a 3D representation of a scene, the scene comprising at least one action paused at a point in time;
and
a plurality of temporal deformations corresponding to transformations of the 3D representation to show the scene at different points in time during the action;
the 3D representation and temporal deformations being generated based on a reference video showing the action occurring in the scene;
and
applying a point of view transformation and at least one temporal deformation, selected from the plurality of temporal deformations, to the 3D representation to generate one or more views of the scene.

4 more elements

Dependencies are inferred from the claim text.

Photorealistic 4D scene generation using video diffusion models

Life of this patent

Photorealistic 4D scene generation using video diffusion models

Life of this patent

Abstract

Claims20