Device and method for controlling a robot to perform a task

A robot controller configured to control a robot to perform a task, the robot controller configured to:

• acquire a target image data element including at least one target image from a perspective of an end-effector of the robot at a target position of the robot in which the robot has performed the task;
• acquire an origin image data element including at least one origin image from the perspective of the end-effector of the robot at an origin position of the robot;
• supply the origin image data element and the target image data element to a machine learning model configured to derive a delta movement between the origin position and the target position;
• control the robot to move according to the delta movement to perform the task;
• train the machine learning model to derive delta movements from pairs of image data elements by collecting a set of primary training elements, each primary training element of the set of preliminary training elements including an image data element including at least one image from the perspective of the end-effector at a respective position and a movement vector for moving the end-effector from the respective position to a desired position;
• generate secondary training elements by choosing, for each secondary training element of the secondary training elements, two of the primary training elements, including the image data elements of the primary training elements into the secondary training element and including a difference between the movement vectors of the primary training elements as a ground truth movement vector;
• and
• train the machine learning model using the secondary training elements in a supervised manner.

A robot, comprising:

• a robot controller configured to control the robot to perform a task, the robot controller configured to:
• acquire a target image data element including at least one target image from a perspective of an end-effector of the robot at a target position of the robot in which the robot has performed the task,
• acquire an origin image data element including at least one origin image from the perspective of the end-effector of the robot at an origin position of the robot,
• supply the origin image data element and the target image data element to a machine learning model configured to derive a delta movement between the origin position and the target position,
• control the robot to move according to the delta movement to perform the task,
• train the machine learning model to derive delta movements from pairs of image data elements by collecting a set of primary training elements, each primary training element of the set of preliminary training elements including an image data element including at least one image from the perspective of the end-effector at a respective position and a movement vector for moving the end-effector from the respective position to a desired position,
• generate secondary training elements by choosing, for each secondary training element of the secondary training elements, two of the primary training elements, including the image data elements of the primary training elements into the secondary training element and including a difference between the movement vectors of the primary training elements as a ground truth movement vector, and
• training the machine learning model using the secondary training elements in a supervised manner;
• and
• an end-effector with at least one camera configured to acquire the at least one origin image.

A non-transitory computer readable medium on which are stored instructions for controlling a robot to perform a task, the instructions, when executed by a processor, causing the processor to perform the following steps:

• acquiring a target image data element including at least one target image from a perspective of an end-effector of the robot at a target position of the robot in which the robot has performed the task;
• acquiring an origin image data element including at least one origin image from the perspective of the end-effector of the robot at an origin position of the robot;
• supplying the origin image data element and the target image data element to a machine learning model configured to derive a delta movement between the origin position and the target position;
• and
• controlling the robot to move according to the delta movement to perform the task;
• training the machine learning model to derive delta movements from pairs of image data elements by collecting a set of primary training elements, each primary training element of the set of preliminary training elements including an image data element including at least one image from the perspective of the end-effector at a respective position and a movement vector for moving the end-effector from the respective position to a desired position;
• generating secondary training elements by choosing, for each secondary training element of the secondary training elements, two of the primary training elements, including the image data elements of the primary training elements into the secondary training element and including a difference between the movement vectors of the primary training elements as a ground truth movement vector;
• and
• training the machine learning model using the secondary training elements in a supervised manner.