Action prediction networks for robotic grasping

A method of training an action prediction network, comprising:

• identifying, by one or more processors, a plurality of training examples generated based on sensor output from one or more robots during a plurality of grasp attempts by the robots,
• each of the training examples including training example input comprising:
• an image for a corresponding instance of time of a corresponding grasp attempt of the grasp attempts, the image capturing a robotic end effector and at least one environmental object at the corresponding instance of time,
• each of the training examples including training example output comprising a grasp success label indicative of success of the corresponding grasp attempt;
• applying, by one or more of the processors, the image to a processing network portion of the action prediction network to generate one or more features of the image,
• wherein the processing network includes one or more convolutional neural networks, a softmax layer, and one or more feed forward layers;
• applying, by the one or more processors, the one or more features of the image to a neural density model portion of the action prediction network to generate a predicted probability density of candidate actions,
• wherein the predicted probability density defines a plurality of candidate actions, each of the candidate actions having a respective probability that implementing the candidate action, and subsequently grasping, will result in a successful grasp of the object,
• wherein each of the plurality of candidate actions indicates at least a respective three-dimensional direction of movement for the grasping end effector, and
• wherein the neural density model portion of the action prediction network is downstream from the processing model portion of the action prediction network;
• determining a loss as a function of the grasp success label and the predicted probability density.

A system comprising:

• a vision sensor viewing an environment;
• a trained action prediction network stored in one or more non-transitory computer readable media;
• at least one processor configured to:
• at a given iteration of visual servoing the a grasping end effector of a robot using the trained action prediction network to grasp the object in the environment:
• apply a current image capturing the grasping end effector of the robot in a current pose and the object in the environment of the robot, wherein the current image is captured by the vision sensor;
• generate, using the trained action prediction network, a predicted probability density of candidate actions,
• wherein the predicted probability density is generated using a neural density model portion of the action prediction network,
• wherein each of the candidate actions indicates at least a respective three-dimensional direction of movement for the grasping end effector, and
• wherein the predicted probability density defines, for each of the candidate actions, a respective probability that implementing the candidate action, and subsequently grasping, will result in a successful grasp of the object;
• select a given action, of the candidate actions, based on the generated predicted probability density;
• generate an end effector command based on the selected action;
• and
• provide the end effector command to one or more actuators of the robot.