Systems and methods for reconstructing a scene in three dimensions from a two-dimensional image

A non-transitory computer-readable medium for reconstructing a scene in three dimensions from a two-dimensional image and storing instructions that when executed by one or more processors cause the one or more processors to:

• process an image using a detection transformer to detect an object in the scene and to generate a first latent vector for the object, a Normalized Object Coordinate Space (NOCS) map of the object, and a depth map for a background portion of the scene;
• process the first latent vector using one or more multilayer perceptrons (MLPs) to produce a second latent vector for the object that represents the object in a differentiable database of object priors, wherein the differentiable database of object priors encodes geometry of the object priors using signed distance fields (SDFs) and appearance of the object priors using luminance fields (LFs);
• recover, from the NOCS map of the object, a partial three-dimensional (3D) shape of the object;
• estimate an initial pose of the object;
• fit an object prior in the differentiable database of object priors to align in geometry and appearance with the partial 3D shape of the object to produce a complete shape of the object and refine the initial pose of the object using a surfel-based differentiable renderer to produce a refined estimated pose of the object;
• generate an editable and re-renderable 3D reconstruction of the scene based, at least in part, on the complete shape of the object, the refined estimated pose of the object, and the depth map for the background portion of the scene;
• and
• control operation of a robot based, at least in part, on the editable and re-renderable 3D reconstruction of the scene.

A method of reconstructing a scene in three dimensions from a two-dimensional image, the method comprising:

• processing an image using a detection transformer to detect an object in the scene and to generate a first latent vector for the object, a Normalized Object Coordinate Space (NOCS) map of the object, and a depth map for a background portion of the scene;
• processing the first latent vector using one or more multilayer perceptrons (MLPs) to produce a second latent vector for the object that represents the object in a differentiable database of object priors, wherein the differentiable database of object priors encodes geometry of the object priors using signed distance fields (SDFs) and appearance of the object priors using luminance fields (LFs);
• recovering, from the NOCS map of the object, a partial three-dimensional (3D) shape of the object;
• estimating an initial pose of the object;
• fitting an object prior in the differentiable database of object priors to align in geometry and appearance with the partial 3D shape of the object to produce a complete shape of the object and refining the initial pose of the object using a surfel-based differentiable renderer to produce a refined estimated pose of the object;
• generating an editable and re-renderable 3D reconstruction of the scene based, at least in part, on the complete shape of the object, the refined estimated pose of the object, and the depth map for the background portion of the scene;
• and
• controlling operation of a robot based, at least in part, on the editable and re-renderable 3D reconstruction of the scene.