Detection and reconstruction of an environment to facilitate robotic interaction with the environment

A non-transitory computer readable medium having stored thereon instructions that, upon execution by a computing device, cause the computing device to perform functions comprising:

• determining a three-dimensional (3D) virtual environment based on data received from one or more sensors, the 3D virtual environment being representative of a physical environment of a robotic manipulator including a plurality of 3D virtual objects corresponding to respective physical objects in the physical environment;
• determining one or more two-dimensional (2D) images of the 3D virtual environment, wherein the one or more 2D images include respective 2D depth maps representative of distances between respective surfaces of the physical objects and a reference plane associated with a perspective of the one or more sensors;
• determining one or more portions of the one or more 2D images corresponding to a given one or more physical objects;
• based on the determined one or more portions of the one or more 2D images and further based on portions of the 2D depth maps associated with the given one or more physical objects, determining 3D models corresponding to respective determined portions of the one or more 2D images;
• based on the determined 3D models, selecting a particular physical object from the given one or more physical objects;
• and
• providing an instruction to the robotic manipulator to move the particular physical object.

A system comprising:

• a robotic manipulator;
• one or more sensors;
• at least one processor;
• and
• data storage comprising instructions executable by the at least one processor to cause the system to perform functions comprising:
• determining a three-dimensional (3D) virtual environment based on data received from the one or more sensors, the 3D virtual environment being representative of a physical environment of the robotic manipulator including a plurality of 3D virtual objects corresponding to respective physical objects in the physical environment;
• determining one or more two-dimensional (2D) images of the 3D virtual environment, wherein the one or more 2D images include respective 2D depth maps representative of distances between respective surfaces of the physical objects and a reference plane associated with a perspective of the one or more sensors;
• determining one or more portions of the one or more 2D images corresponding to a given one or more physical objects;
• based on the determined one or more portions of the one or more 2D images and further based on portions of the 2D depth maps associated with the given one or more physical objects, determining 3D models corresponding to respective determined portions of the one or more 2D images;
• based on the determined 3D models, selecting a particular physical object from the given one or more physical objects;
• and
• providing an instruction to the robotic manipulator to move the particular physical object.