Device and method for training a machine-learning model for determining a grasp of a multi-finger gripper for manipulating an object

A data processing device configured to train a machine-learning model to determine grasps of a multi-finger gripper for manipulating an object, the data processing device comprising a processing system that includes at least one processor, wherein the processing system is programmed to:

• generate, for a scene that includes an instance of the object, a three-dimensional point cloud representation of the scene including the object;
• obtain, from the three-dimensional point cloud representation, a subset of distinct three-dimensional points that collectively represent three-dimensional contours of an exterior surface of the object;
• identify, from the subset of distinct three-dimensional points that have been obtained from the three-dimensional point cloud representation, a plurality of distinct surface points located at different respective three-dimensional positions on the exterior surface of the object, wherein at least a subset of the plurality of distinct surface points each is associated with a respective set of one or more of a plurality of ground-truth grasps;
• determine, by the machine-learning model, a plurality of predicted grasps for the plurality of distinct surface points, wherein each respective one of the plurality of predicted grasps:
• is associated with a respective surface point of the plurality of distinct surface points;
• and
• includes a predicted palm pose and predicted finger-joint configurations of the multi-finger gripper;
• determine, for each respective surface point of the subset of the plurality of distinct surface points, a respective grasp-loss value representing a difference between a respective predicted grasp associated with the respective surface point and a selected ground-truth grasp from the respective set associated with the respective surface point, wherein the grasp-loss values are based on a collision loss and/or a grasp stability loss;
• combine the grasp-loss values for the plurality of distinct surface points to determine an overall training loss for the scene;
• and
• adjust parameters of the machine-learning model to reduce the overall training loss.

A non-transitory computer-readable medium on which are stored instructions for training a machine-learning model to determine grasps of a multi-finger gripper for manipulating an object, the instructions, when executed by a processor, causing the processor to perform the following steps:

• generating, for a scene that includes an instance of the object, a three-dimensional point cloud representation of the scene including the object;
• obtaining, from the three-dimensional point cloud representation, a subset of distinct three-dimensional points that collectively represent three-dimensional contours of an exterior surface of the object;
• identifying, from the subset of distinct three-dimensional points that have been obtained from the three-dimensional point cloud representation, a plurality of distinct surface points located at different respective three-dimensional positions on the exterior surface of the object, wherein at least a subset of the plurality of distinct surface points each is associated with a respective set of one or more of a plurality of ground-truth grasps;
• determining, by the machine-learning model, a plurality of predicted grasps for the plurality of distinct surface points, wherein each respective one of the plurality of predicted grasps:
• is associated with a respective surface point of the plurality of distinct surface points;
• and
• includes a predicted palm pose and predicted finger-joint configurations of the multi-finger gripper;
• determining, for each respective surface point of the subset of the plurality of distinct surface points, a respective grasp-loss value representing a difference between a respective predicted grasp associated with the respective surface point and a selected ground-truth grasp from the respective set associated with the respective surface point, wherein the grasp-loss values are based on a collision loss and/or a grasp stability loss;
• combining the grasp-loss values for the plurality of distinct surface points to determine an overall training loss for the scene;
• and
• adjusting parameters of the machine-learning model to reduce the overall training loss.