User Interface for Supervised Autonomous Grasping

A robot comprising:

• a body;
• a robotic manipulator coupled to the body, the robotic manipulator comprising an end-effector configured to grasp objects within an environment about the robot;
• data processing hardware in communication with the robotic manipulator;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
• receiving sensor data for a space within an environment about the robot;
• receiving, from a user interface (UI) in communication with the data processing hardware, a user input indicating a user-selection of a location within a two-dimensional (2D) representation of the space, the location corresponding to a position of a target object within the space;
• receiving, from the UI, a plurality of grasping inputs designating an orientation and a translation for the end-effector of the robotic manipulator to grasp the target object;
• generating a three-dimensional (3D) location of the target object based on the received sensor data and the location corresponding to the user input;
• and
• instructing the end-effector of the robotic manipulator to grasp the target object using the generated 3D location and the plurality of grasping inputs designating the orientation and the translation of the end-effector of the robotic manipulator.