Device and method for training a machine learning model for generating descriptor images for images of objects

A method for controlling a robot to pick up or process an object, comprising the following steps:

• training a machine learning model for generating descriptor images for images of one or more objects, including:
• recording multiple camera images, each of the camera images showing one or more objects,
• for each camera image of the camera images:
• generating one or more augmented versions of the camera image by applying a respective augmentation to the camera image for each augmented version of the camera image, wherein the augmentation includes a change in position of pixel values of the camera image,
• generating one or more pairs of training images which each include the camera image and an augmentation of the camera image or two augmented versions of the camera image, and
• determining, for each pair of training images and from pixels of the pair of training image, according to the change in position of pixel values included in the respective augmentation with which the augmented version was generated, if the pair of training images includes an augmented version of the camera image, or, according to the changes in position of pixel values included in the two respective augmentations with which the augmented versions were generated, if the pair of training images includes two augmented versions of the camera image, pixels of the pair of training images which correspond to one another and pixels of the pair of training images which do not correspond to one another, and
• training the machine learning model with contrastive loss using the pairs of training images, wherein descriptor values which are generated by the machine learning model for the pixels which correspond to one another are used as positive pairs and descriptor values which are generated by the machine learning model for the pixels which do not correspond to one another are used as negative pairs;
• recording a camera image which shows the object in a current control scenario;
• feeding the camera image which shows the object to the machine learning model to generate a descriptor image;
• determining a position of a location for picking up or processing the object in the current control scenario from the descriptor image;
• and
• controlling the robot according to the determined position.

A control unit configured to train a machine learning model for generating descriptor images for images of one or more objects, the control unit configured to:

• record multiple camera images, each of the camera images showing one or more objects;
• for each camera image of the camera images:
• generate one or more augmented versions of the camera image by applying a respective augmentation to the camera image for each augmented version of the camera image, wherein the augmentation includes a change in position of pixel values of the camera image,
• generate one or more pairs of training images which each include the camera image and an augmentation of the camera image or two augmented versions of the camera image, and
• determine, for each pair of training images and from pixels of the pair of training image, according to the change in position of pixel values included in the respective augmentation with which the augmented version was generated, if the pair of training images includes an augmented version of the camera image, or, according to the changes in position of pixel values included in the two respective augmentations with which the augmented versions were generated, if the pair of training images includes two augmented versions of the camera image, pixels of the pair of training images which correspond to one another and pixels of the pair of training images which do not correspond to one another;
• and
• train the machine learning model with contrastive loss using the pairs of training images, wherein descriptor values which are generated by the machine learning model for the pixels which correspond to one another are used as positive pairs and descriptor values which are generated by the machine learning model for the pixels which do not correspond to one another are used as negative pairs.

A non-transitory computer-readable medium on which are stored instructions for training a machine learning model for generating descriptor images for images of one or more objects, the instructions, when executed by a processor, causing the processor to perform the following steps:

• recording multiple camera images, each of the camera images showing one or more objects;
• for each camera image of the camera images:
• generating one or more augmented versions of the camera image by applying a respective augmentation to the camera image for each augmented version of the camera image, wherein the augmentation includes a change in position of pixel values of the camera image,
• generating one or more pairs of training images which each include the camera image and an augmentation of the camera image or two augmented versions of the camera image, and
• determining, for each pair of training images and from pixels of the pair of training image, according to the change in position of pixel values included in the respective augmentation with which the augmented version was generated, if the pair of training images includes an augmented version of the camera image, or, according to the changes in position of pixel values included in the two respective augmentations with which the augmented versions were generated, if the pair of training images includes two augmented versions of the camera image, pixels of the pair of training images which correspond to one another and pixels of the pair of training images which do not correspond to one another;
• and
• training the machine learning model with contrastive loss using the pairs of training images, wherein descriptor values which are generated by the machine learning model for the pixels which correspond to one another are used as positive pairs and descriptor values which are generated by the machine learning model for the pixels which do not correspond to one another are used as negative pairs.