Continuous updating of plan for robotic object manipulation based on received sensor data

A system, comprising:

• a robotic manipulator;
• at least one sensor on the robotic manipulator;
• and
• a control system configured to:
• determine a virtual environment based on sensor data received from the at least one sensor on the robotic manipulator, the virtual environment representing a physical environment containing a plurality of physical objects;
• develop a plan, based on the virtual environment, to cause the robotic manipulator to move one or more of the physical objects in the physical environment;
• cause the robotic manipulator to pick up at least one object of the plurality of physical objects with the robotic manipulator, move the at least one object to a drop-off location, and drop off the at least one object at the drop-off location according to the plan;
• after the robotic manipulator drops off the at least one object at the drop-off location, cause the robotic manipulator to perform a scan of an area vacated by the at least one object by moving the robotic manipulator while receiving updated sensor data from the one or more sensors on the robotic manipulator, wherein the updated sensor data is indicative of at least one other physical object previously obscured by the at least one object;
• modify the virtual environment based on the updated sensor data;
• determine one or more modifications to the plan based on the modified virtual environment;
• and
• cause the robotic manipulator to perform another action according to the modified plan.

A non-transitory computer readable medium having stored therein instructions, that when executed by a computing system, cause the computing system to perform functions comprising:

• determining a virtual environment by one or more processors based on sensor data received from one or more sensors on a robotic manipulator, the virtual environment representing a physical environment containing a plurality of physical objects;
• developing a plan, based on the virtual environment, to cause the robotic manipulator to move one or more of the physical objects in the physical environment;
• causing the robotic manipulator to pick up at least one object of the plurality of physical objects with the robotic manipulator, move the at least one object to a drop-off location, and drop off the at least one object at the drop-off location according to the plan;
• after the robotic manipulator drops off the at least one object at the drop-off location, causing the robotic manipulator to perform a scan of an area vacated by the at least one object by moving the robotic manipulator while receiving updated sensor data from the one or more sensors on the robotic manipulator, wherein the updated sensor data is indicative of at least one other physical object previously obscured by the at least one object;
• modifying the virtual environment based on the updated sensor data;
• determining one or more modifications to the plan based on the modified virtual environment;
• and
• causing the robotic manipulator to perform another action according to the modified plan.