US 2024/0193855 A1 · SPATIALLY DISENTANGLED GENERATIVE RADIANCE FIELDS FOR CONTROLLABLE 3D-AWARE SCENE SYNTHESIS

Abstract

A 3D-aware generative model for high-quality and controllable scene synthesis uses an abstract object-level representation (i.e., 3D bounding boxes without semantic annotation) as the scene layout prior, which is simple to obtain, general to describe various scene contents, and yet informative to disentangle objects and background. An overall layout for the scene is identified and then each object is located in the layout to facilitate the scene composition process. The object-level representation serves as an intuitive user control for scene editing. Based on such a prior, the system spatially disentangles the whole scene into object-centric generative radiance fields by learning on only 2D images with global-local discrimination. Once the model is trained, users can generate and edit a scene by explicitly controlling the camera and the layout of objects' bounding boxes.

Claims20

Dependent claims

A method of generating individual objects and a background of a three-dimensional (3D) scene, the method comprising:

receiving, by an object generator, a data set of input images and 3D bounding boxes of objects in the input images;
receiving, by the object generator, a layout prior of a scene in the input images;
generating, by the object generator, an object for each 3D bounding box;
generating, by a background generator, a background of the scene;
and
generating, by a volume renderer, a version of the scene from the objects for each 3D bounding box and the background of the scene.

3 more elements

A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a processor cause the processor to generate individual objects and a background of a three-dimensional (3D) scene by performing operations comprising:

receiving a data set of input images and 3D bounding boxes of objects in the input images;
receiving a layout prior of a scene in the input images;
generating an object for each 3D bounding box;
generating a background of the scene;
and
generating a version of the scene from the objects for each 3D bounding box and the background of the scene.

3 more elements

Dependencies are inferred from the claim text.

Spatially disentangled generative radiance fields for controllable 3d-aware scene synthesis

Life of this patent

Spatially disentangled generative radiance fields for controllable 3d-aware scene synthesis

Life of this patent

Abstract

Claims20