Device and method for training a machine learning model for generating descriptor images for images of objects

A method for controlling a robot for picking up or processing an object, comprising:

• training a machine learning model including:
• forming pairs of images, each image pair of the pairs of images including a first image and a second image, which show the one or the multiple objects from different perspectives;
• generating, for each image pair, using the machine learning model, a first descriptor image for the first image of the image pair, which assigns descriptors to points of the one or multiple objects shown in the first image of the image pair, and a second descriptor image for the second image of the image pair, which assigns descriptors to points of the one or multiple objects shown in the second image of the image pair;
• sampling, for each image pair, descriptor pairs, which each include a first descriptor from the first descriptor image and a second descriptor from the second descriptor image, which are assigned to the same point;
• adapting the machine learning method for reducing a loss, which includes for each sampled descriptor pair the ratio of the distance according to a distance measure between the first descriptor and the second descriptor to the sum of all distances according to the distance measure between the first descriptor and the descriptors of the second descriptor image, which appear in the sampled descriptor pairs;
• wherein the method is performed without object masks and further comprises the following steps:
• recording the one or multiple objects in camera images;
• obtaining additional images by augmenting at least a portion of the camera images, and forming the pair of images from the camera images and additional images, each of the pairs of images including a camera image and a camera image obtained by augmentation, the augmentation including one or multiple of:
• resizing and cropping, perspective and affine distortion, horizontal and vertical mirroring, rotation, addition of blurring, addition of color noise and conversion to grayscale;
• recording a camera image which shows the object in an instantaneous control scenario;
• feeding the camera image to the machine learning model for generating a descriptor image;
• ascertaining the position of a point for picking up or processing the object in the instantaneous control scenario from the descriptor image;
• and
• controlling the robot according to the ascertained position.

A control unit configured to train a machine learning model for generating descriptor images for images of one or of multiple objects, the control unit configured to:

• form pairs of images, each image pair of the pairs of images including a first image and a second image, which show the one or the multiple objects from different perspectives;
• generate, for each image pair, using the machine learning model, a first descriptor image for the first image of the image pair, which assigns descriptors to points of the one or multiple objects shown in the first image of the image pair, and a second descriptor image for the second image of the image pair, which assigns descriptors to points of the one or multiple objects shown in the second image of the image pair;
• sample, for each image pair, descriptor pairs, which each include a first descriptor from the first descriptor image and a second descriptor from the second descriptor image, which are assigned to the same point;
• adapt the machine learning method for reducing a loss, which includes for each sampled descriptor pair the ratio of the distance according to a distance measure between the first descriptor and the second descriptor to the sum of all distances according to the distance measure between the first descriptor and the descriptors of the second descriptor image, which appear in the sampled descriptor pairs;
• wherein the control unit operates without using object masks and is further configured to:
• record the one or multiple objects in camera images;
• obtain additional images by augmenting at least a portion of the camera images, and forming the pair of images from the camera images and additional images, each of the pairs of images including a camera image and a camera image obtained by augmentation, the augmentation including one or multiple of:
• resizing and cropping, perspective and affine distortion, horizontal and vertical mirroring, rotation, addition of blurring, addition of color noise and conversion to grayscale.

A non-transitory computer-readable memory medium on which is stored a computer program for training a machine learning model for generating descriptor images for images of one or of multiple objects, the computer program, when executed by a computer, causing the computer to perform the following steps:

• forming pairs of images, each image pair of the pairs of images including a first image and a second image, which show the one or the multiple objects from different perspectives;
• generating, for each image pair, using the machine learning model, a first descriptor image for the first image of the image pair, which assigns descriptors to points of the one or multiple objects shown in the first image of the image pair, and a second descriptor image for the second image of the image pair, which assigns descriptors to points of the one or multiple objects shown in the second image of the image pair;
• sampling, for each image pair, descriptor pairs, which each include a first descriptor from the first descriptor image and a second descriptor from the second descriptor image, which are assigned to the same point;
• adapting the machine learning method for reducing a loss, which includes for each sampled descriptor pair the ratio of the distance according to a distance measure between the first descriptor and the second descriptor to the sum of all distances according to the distance measure between the first descriptor and the descriptors of the second descriptor image, which appear in the sampled descriptor pairs;
• wherein the computer program, when executed by the computer, further causes the computer to operate without object masks and perform the following steps:
• recording the one or multiple objects in camera images;
• obtaining additional images by augmenting at least a portion of the camera images, and forming the pair of images from the camera images and additional images, each of the pairs of images including a camera image and a camera image obtained by augmentation, the augmentation including one or multiple of:
• resizing and cropping, perspective and affine distortion, horizontal and vertical mirroring, rotation, addition of blurring, addition of color noise and conversion to grayscale.