Learning across 2d and 3d pipelines for improved object detection

A system comprising:

• one or more non-transitory computer-readable storage media including instructions;
• and
• one or more processors coupled to the one or more non-transitory storage media, the one or more processors configured to execute the instructions to:
• access a training sample comprising (1) an image of a scene including an object, (2) depth measurements of the scene, and (3) a predetermined 3D position of an object in the scene;
• train a 3D-detection model for detecting 3D positions of objects based on at least the depth measurement of the scene and the predetermined 3D position of the object;
• and
• train a 2D-detection model for detecting 2D positions of objects within images by:
• generate an estimated 2D position of the object within the image by processing the image using the 2D-detection model;
• determine a subset of the depth measurements that correspond to the object based on the estimated 2D position of the object within the image and a viewpoint from which the image is captured;
• generate an estimated 3D position of the object based on the subset of the depth measurements;
• and
• update the 2D-detection model based on a comparison between the estimated 3D position of the object and the predetermined 3D position of the object.

A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of a computing system, cause the one or more processors to:

• access a training sample comprising (1) an image of a scene including an object, (2) depth measurements of the scene, and (3) a predetermined 3D position of an object in the scene;
• train a 3D-detection model for detecting 3D positions of objects based on at least the depth measurement of the scene and the predetermined 3D position of the object;
• and
• train a 2D-detection model for detecting 2D positions of objects within images by:
• generate an estimated 2D position of the object within the image by processing the image using the 2D-detection model;
• determine a subset of the depth measurements that correspond to the object based on the estimated 2D position of the object within the image and a viewpoint from which the image is captured;
• generate an estimated 3D position of the object based on the subset of the depth measurements;
• and
• update the 2D-detection model based on a comparison between the estimated 3D position of the object and the predetermined 3D position of the object.