Methods and systems for automated three-dimensional object detection and extraction

A system for extracting a three-dimensional (3D) object from a 3D environment comprising a mesh of triangles, the system comprising:

• a processor;
• and
• a non-transitory computer-readable storage medium having instructions stored thereon that, when executed, causes the processor to:
• segment an initial two-dimensional (2D) scene of the 3D environment obtained from an initial viewpoint, thereby identifying a plurality of object instances within the initial 2D scene;
• map a selected one of the object instances to an initial subset of the triangles underlying the selected one of the object instances, wherein the selected one of the object instances is to be extracted, wherein said mapping comprises performing a matching operation between a triangle identification map and an initial segmentation mask corresponding to the object instance to be extracted to identify the subset of triangles, the matching operation comprising:
• retrieving the initial segmentation mask corresponding to the object instance to be extracted;
• retrieving the triangle identification map, wherein the triangle identification map corresponds to the 2D scene captured from said initial viewpoint;
• and
• correlating said triangle identification map to said segmentation mask corresponding to the object instance to be extracted to identify the initial subset of triangles;
• generate a plurality of additional viewpoints around the selected one of object instances within the 3D environment, and for each one of the plurality of additional viewpoints:
• generate a further 2D scene based on the 3D environment, the further 2D scene generated from the additional viewpoint, the additional viewpoint positioned based on at least one other additional viewpoint in the plurality of additional viewpoints;
• segment the further 2D scene, thereby identifying candidate object instances present in the further 2D scene;
• identify a given one of the candidate object instances as best matching the object instance to be extracted, thereby obtaining a best matching candidate object instance;
• identify an additional subset of the triangles underlying the best matching candidate object instance;
• and
• aggregate the additional subset of the triangles to the initial subset of the triangles, thereby obtaining an updated subset of the triangles;
• and
• output the updated subset of the triangles, thereby extracting the object instance to be extracted.

An apparatus for extracting a three-dimensional (3D) object from a 3D environment comprising a mesh of triangles, the apparatus comprising:

• a memory, comprising:
• an instance segmentation neural network model configured to segment an initial two-dimensional (2D) scene of the 3D environment obtained from an initial viewpoint, thereby identifying a plurality of object instances within the initial 2D scene;
• and
• a mapping operation for mapping a selected one of the object instances to an initial subset of the triangles underlying the selected one of the object instances, wherein the selected one of the object instances is to be extracted, wherein the mapping operation is configured to identify the subset of triangles using a matching operation between a triangle identification map and a segmentation mask of the selected object instance, the matching operation comprising:
• retrieving the initial segmentation mask corresponding to the object instance to be extracted;
• retrieving the triangle identification map, wherein the triangle identification map corresponds to the 2D scene captured from said initial viewpoint;
• and
• correlating said triangle identification map to said segmentation mask corresponding to the object instance to be extracted to identify the initial subset of triangles;
• and
• a processor in communication with the memory, the processor configured to generate a plurality of additional viewpoints around the selected one of object instances within the 3D environment, and for each additional viewpoint of the plurality of additional viewpoints:
• generate a further 2D scene based on the 3D environment, the further 2D scene generated from the additional viewpoint, the additional viewpoint positioned based on the at least one other additional viewpoint in the plurality of additional viewpoints;
• direct the instance segmentation neural network model to segment the further 2D scene, thereby identifying candidate object instances present in the further 2D scene;
• identify a given one of the candidate object instances as best matching the object instance to be extracted, thereby obtaining a best matching candidate object instance;
• direct the mapping operation to identify an additional subset of the triangles underlying the best matching candidate object instance;
• aggregate the additional subset of the triangles to the initial subset of the triangles, thereby obtaining an updated subset of the triangles;
• and
• output the updated subset of the triangles, thereby extracting the object instance to be extracted.