Robotic grasping prediction using neural networks and geometry aware object representation

A method implemented by one or more processors, comprising:

• identifying a current image captured by a vision sensor of a robot, the current image capturing an end effector of the robot and an object to be grasped by the robot using the end effector;
• generating an encoding of:
• the current image, or an additional image that captures the object;
• wherein generating the encoding comprises processing the current image or the additional image using a trained neural network encoder, and wherein the generated encoding encodes geometry features of the object;
• applying, as one or more inputs to a prediction network:
• the current image and the encoding, wherein the prediction network is a trained neural network model;
• generating a prediction using the prediction network based on the one or more inputs;
• determining that the outcome prediction satisfies one or more criteria;
• and
• in response to the prediction satisfying one or more criteria, providing, to one or more actuators of the robot, control commands that are based on the prediction.

A robot, comprising:

• a vision sensor viewing an environment;
• a trained neural network encoder and a trained grasp outcome prediction network stored in one or more non-transitory computer readable media;
• one or more processors processor configured to:
• identify a current image captured by the vision sensor, the current image capturing an object to be grasped by the robot using the end effector;
• determine a candidate grasp pose of the end effector;
• generate an encoding of:
• the current image, or an additional image that captures the object;
• wherein in generating the encoding one or more of the processors are to generate the encoding by processing the current image or the additional image using the trained neural network encoder, and wherein the generated encoding encodes geometry features of the object;
• apply, as one or more inputs to the trained grasp outcome prediction network:
• the current image, the candidate grasp pose, and the encoding;
• generate, using the trained grasp outcome prediction network based on the one or more inputs, a grasp outcome prediction;
• and
• provide control commands to one or more actuators of the robot based on the grasp outcome prediction.