Grasping of an object by a robot based on grasp strategy determined using machine learning model(s)

A method implemented by one or more processors,

• comprising:
• applying vision data as input to at least one trained machine learning model, the vision data being generated by one or more vision components of a robot and capturing features of an object in an environment of the robot and additional features of an alternative object in the environment;
• processing the vision data using the at least one trained machine learning model to generate output, of the machine learning model, defining:
• a spatial region for interacting with the object to grasp the object, the spatial region being a bounding area that encompasses a portion of the vision data corresponding to the object, and
• a semantic indication for the spatial region,
• an alternative spatial region for interacting with the alternative object, the alternative spatial region being an alternative bounding area that encompasses an alternative portion of the vision data corresponding to the alternative object, and
• an alternative semantic indication for the additional spatial region;
• selecting the spatial region based on the spatial region corresponding to the object and the object being selected for grasping;
• selecting, based on the semantic indication, a particular grasp strategy of a plurality of candidate grasp strategies, wherein selecting the particular grasp strategy based on the semantic indication is based on the semantic indication being for the spatial region and the spatial region being selected;
• determining an end effector pose for interacting with the object to grasp the object, wherein determining the end effector pose is based on the spatial region defined by the output and is based on the particular grasp strategy selected based on the semantic indication defined by the output;
• and
• providing, to actuators of the robot, commands that cause an end effector of the robot to traverse to the end effector pose in association with attempting a grasp of the object.