Synergies between pick and place: task-aware grasp estimation

A method for controlling a robot including a manipulator, the method comprising:

• determining three-dimensional (3D) geometry information about a target object based on an image of the target object;
• determining 3D geometry information about a scene in which the target object is to be placed based on at least one image of the scene;
• determining a plurality of candidate placement directions for placing the target object in the scene based on the 3D geometry information about the scene;
• obtaining affordance information by providing the 3D geometry information about the target object and information about the plurality of candidate placement directions to at least one neural network model;
• commanding the robot to grasp the target object using the manipulator according to a grasp orientation corresponding to the affordance information;
• and
• commanding the robot to position the manipulator according to a placement direction corresponding to the affordance information in order to place the target object at a location in the scene.

A non-transitory computer-readable medium configured to store instructions which, when executed by at least one processor of a device for controlling a robot including a manipulator, cause the at least one processor to:

• determine three-dimensional (3D) geometry information about a target object based on an image of the target object;
• determine 3D geometry information about a scene in which the target object is to be placed based on at least one image of the scene;
• determine a plurality of candidate placement directions for placing the target object in the scene based on the 3D geometry information about the scene;
• obtain affordance information by providing the 3D geometry information about the target object and information about the plurality of candidate placement directions to at least one neural network model;
• command the robot to grasp the target object using the manipulator according to a grasp orientation corresponding to the affordance information;
• and
• command the robot to position the manipulator according to a placement direction corresponding to the affordance information in order to place the target object at a location in the scene.