3-dimensional scene analysis for augmented reality operations

A system for 3-Dimensional (3D) scene analysis, the system comprising:

• a 3D reconstruction circuit to receive a plurality of 3D image frames of a scene, each frame comprising a red-green-blue (RGB) image frame comprising color pixels and a depth map frame comprising depth pixels, wherein each of the 3D image frames is associated with a pose of a depth camera that generated the 3D image frames, the 3D reconstruction circuit further to project the depth pixels into points in a global coordinate system based on the camera pose and accumulate the projected points into a 3D reconstruction of the scene;
• an object detection circuit to:
• detect objects and associated locations in the scene, for each 3D image frame, based on the camera pose, the 3D reconstruction, the RGB image frame and the depth map frame;
• associate a label with the detected object;
• calculate a 2-Dimensional (2D) bounding box containing the detected object, and a 3D location of the center of the 2D bounding box;
• and
• match the detected object to an existing object boundary set created from a previously received 3D image frame, the match based on the label and the 3D location of the center of the 2D bounding box;
• a 3D segmentation circuit to segment each of the detected objects in the scene, the segmented objects comprising the points of the 3D reconstruction corresponding to contours of the associated detected object;
• and
• a 3D registration circuit to register the segmented objects to a 3D model of the associated detected object to determine an alignment of the detected object in the scene.

At least one non-transitory computer readable storage medium having instructions encoded thereon that, when executed by one or more processors, result in the following operations for 3-Dimensional (3D) scene analysis, the operations comprising:

• receiving a plurality of 3D image frames of a scene, each frame comprising a red-green-blue (RGB) image frame comprising color pixels and a depth map frame comprising depth pixels, wherein each of the 3D image frames is associated with a pose of a depth camera that generated the 3D image frames;
• projecting the depth pixels into points in a global coordinate system based on the camera pose;
• accumulating the projected points into a 3D reconstruction of the scene;
• detecting objects and associated locations in the scene, for each 3D image frame, based on the camera pose, the 3D reconstruction, the RGB image frame and the depth map frame;
• associating a label with the detected object;
• calculating a 2-Dimensional (2D) bounding box containing the detected object, and a 3D location of the center of the 2D bounding box;
• matching the detected object to an existing object boundary set created from a previously received 3D image frame, the matching based on the label and the 3D location of the center of the 2D bounding box;
• segmenting each of the detected objects in the scene, the segmented objects comprising the points of the 3D reconstruction corresponding to contours of the associated detected object;
• and
• registering the segmented objects to a 3D model of the associated detected object to determine an alignment of the detected object in the scene.