US 2026/0311432 A1 · GENERATING 3D REPRESENTATIONS OF PHYSICAL ENVIRONMENTS USING KEY FRAMES

Abstract

Various implementations disclosed herein include devices, systems, and methods that use frames of sensor data to create key frames that are used to provide a 3D representation of a physical environment (e.g., a 3D mesh, planar representations, object bounding boxes, a combo of these, etc.). Some implementations use key frames (e.g., bundles of 2D image data deemed to be representative or useful for reconstruction/3D representation generation). Such key frames may be created by evaluating multiple frames of sensor data and determining whether to convert a given frame into a key frame or fuse it, e.g., fuse it with a nearest existing key frame. Such fusing may reduce loss of information which would occur if frames not selected as key frames are simply discarded.

Claims19

Dependent claims

A method comprising:

at a device having a processor:
obtaining a plurality of frames of sensor data of a physical environment, the frames comprising sensor data and frame-to-frame motion data corresponding to device poses associated with the frames;
generating a 3D representation of the physical environment based on the plurality of frames;
receiving correction data corresponding to a correction associated with the device poses;
determining a global transformation and a change to a non-rigid component based on the correction data;
and
updating the 3D representation by:
applying a transform based on the global transformation to update the 3D representation;
and
regenerating a subset of the 3D representation based on the change to the non-rigid component.

7 more elements

Dependencies are inferred from the claim text.

Generating 3d representations of physical environments using key frames

Life of this patent

Generating 3d representations of physical environments using key frames

Life of this patent

Abstract

Claims19