Supervised autonomous grasping

A robot comprising:

• a body;
• a plurality of legs coupled to the body;
• an end-effector configured to grasp objects within an environment about the robot;
• data processing hardware in communication with the end-effector;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
• receiving a user input selecting a target object for grasping by an end-effector of the robot;
• receiving initial sensor data including information about the target object;
• determining an initial grasp geometry for the end effector to grasp the target object;
• instructing the end-effector to grasp the target object based on the initial grasp geometry;
• receiving updated sensor data in response to the end-effector executing the initial grasp geometry, the updated sensor data including additional information about the target object indicating that a foreign object is present that obstructs the end-effector from grasping the target object based on the initial grasp geometry, the foreign object not included in the initial sensor data;
• modifying the initial grasp geometry based on the additional information about the target object such that the foreign object does not obstruct the end-effector from grasping the target object based on the modified initial grasp geometry;
• and
• controlling the end-effector to grasp the target object based on the modified initial grasp geometry.

A non-transitory computer-readable medium having stored thereon instructions that, when executed by a data processing hardware, cause the data processing hardware to:

• receive a user input selecting a target object for grasping by an end-effector of the robot;
• receiving initial sensor data including information about the target object;
• determine an initial grasp geometry for the end effector to grasp the target object;
• instruct the end-effector to grasp the target object based on the initial grasp geometry;
• receive updated sensor data in response to the end-effector executing the initial grasp geometry, the updated sensor data including additional information about the target object indicating that a foreign object is present that obstructs the end-effector from grasping the target object based on the initial grasp geometry, the foreign object not included in the initial sensor data;
• modify the initial grasp geometry based on the additional information about the target object such that the foreign object does not obstruct the end-effector from grasping the target object based on the modified initial grasp geometry;
• and
• control the end-effector to grasp the target object based on the modified initial grasp geometry.