Kinesthetic teaching of grasp parameters for grasping of objects by a grasping end effector of a robot

A method implemented by one or more processors, comprising:

• identifying vision sensor data generated by at least one vision sensor of a robot in an environment, the vision sensor having an object in its field of view, and the robot having a grasping end effector;
• determining, based on the vision sensor data, an object model that conforms to the object;
• determining, based on the vision sensor data, an object model pose of the object model, the object model pose indicating the object pose of the object in the environment;
• identifying a stored association of the object model to:
• one or more end effector poses defined relative to the object model, and
• at least one force measure;
• converting the end effector poses, defined relative to the object model, to target poses for the grasping end effector, wherein the converting is based on the determined object model pose that conforms to the additional object pose of the additional object in the environment;
• determining, based on target poses, and the at least one force measure:
• control commands to provide to one or more actuators of the robot to attempt a grasp of the object, the control commands including commands determined to cause the grasping end effector to achieve the target poses in the environment and to cause the end effector to impart a force that is based on the force measure, the force being imparted through translation of an entirety of the grasping end effector as a result of movement of one or more actuators that are upstream of the grasping end effector;
• and
• providing the control commands to the one or more actuators.

A system, comprising:

• a vision sensor viewing an environment of a robot and generating vision sensor data;
• one or more processors associated with the robot, the one or more processors configured to:
• determine, based on the vision sensor data, an object model for an object in an environment of a robot and an object model pose of the object model that conforms to an object pose of the object in the environment;
• identify sensor data generated by one or more sensors of the robot during physical manipulation of the robot, the physical manipulation being by a user to cause a grasping end effector of the robot to perform at least part of a grasp sequence for grasping of the object in the environment;
• determine, based on the sensor data and the object model pose for the object model, one or more end effector poses of the grasping end effector, each of the end effector poses defining a pose of the grasping end effector relative to the object model during the grasp sequence;
• determining, based on the sensor data, at least one translational force measure indicating force applied by the robot to the object during at least some of the grasp sequence, the force being a result of translation of an entirety of the grasping end effector, and the translation of the entirety of the grasping end effector being a result of movement of one or more actuators that are upstream of the grasping end effector;
• and
• store, in one or more computer readable media, an association of the object model to:
• the end effector poses defined relative to the object model, and
• the translational force measure.