Device and method for training a neural network for controlling a robot for an inserting task

A method for controlling a robot to insert an object into an insertion, comprising:

• controlling the robot to hold the object;
• generating an estimate of a target position to insert the object into the insertion;
• controlling the robot to move to the estimated target position;
• taking a camera image using a camera after having controlled the robot to move to the estimated target position;
• feeding the camera image to the neural network trained by:
• controlling the robot to hold the object,
• bringing the robot into a first target position in which the object is inserted in the insertion,
• after bringing the robot into the first target position, for each first position of a plurality of first positions different from the first target position:
• controlling the robot to move away from the first target position to the first position,
• taking a first camera image by the camera, and
• labelling the first camera image by a movement vector to move back from the first position to the first target position and by an angle for correcting an orientation according to a difference between an insertion orientation and a current orientation,
• generating, for at least one of the camera images, an additional camera image by at least one of:
• colour jitter, conversion to grey scale, cropping, shifting and filtering of the camera image, and
• labelling the additional camera image with the movement vector with which the camera image is labelled, and
• training the neural network using the labelled first camera images and the labelled additional camera images;
• and
• controlling the robot to move according to a movement vector derived by the neural network from the camera image.

A robot, comprising:

• a camera mounted on the robot adapted to provide camera images;
• and
• a controller configured to implement a neural network and to control a robot to insert an object into an insertion, the controller configured to:
• control the robot to hold the object;
• generate an estimate of a target position to insert the object into the insertion;
• control the robot to move to the estimated target position;
• take a camera image using a camera after having controlled the robot to move to the estimated target position;
• feed the camera image to the neural network trained by:
• controlling the robot to hold the object,
• bringing the robot into a first target position in which the object is inserted in the insertion,
• after bringing the robot into the first target position, for each first position of a plurality of first positions different from the first target position:
• controlling the robot to move away from the first target position to the first position,
• taking a first camera image by the camera,
• labelling the first camera image by a movement vector to move back from the first position to the first target position and by an angle for correcting an orientation according to a difference between an insertion orientation and a current orientation,
• generating, for at least one of the camera images, an additional camera image by at least one of:
• colour jitter, conversion to grey scale, cropping, shifting and filtering of the camera image, and
• labelling the additional camera image with the movement vector with which the camera image is labelled, and
• training the neural network using the labelled first camera images and the labelled additional camera images;
• and
• control the robot to move according to a movement vector derived by the neural network from the camera image.

A non-transitory computer readable medium on which are stored instructions for training a neural network to derive, from an image of a camera mounted on a robot, a movement vector for the robot to insert an object into an insertion, the instruction, when executed by a processor, causing the processor to perform the following steps:

• controlling the robot to hold the object;
• bringing the robot into a target position in which the object is inserted in the insertion;
• after bringing the robot into the target position, for each position of a plurality of positions different from the target position:
• controlling the robot to move away from the target position to the position,
• taking a camera image by the camera,
• labelling the camera image by a movement vector to move back from the position to the target position and by an angle for correcting an orientation according to a difference between an insertion orientation and a current orientation,
• generating, for at least one of the camera images, an additional camera image by at least one of:
• colour jitter, conversion to grey scale, cropping, shifting and filtering of the camera image, and
• labelling the additional camera image with the movement vector with which the camera image is labelled, and
• training the neural network using the labelled camera images and the labelled additional images.

A non-transitory computer readable medium on which are stored instructions for controlling a robot to insert an object into an insertion, the instructions, when executed by a processor, causing the processor to perform the following steps:

• controlling the robot to hold the object;
• generating an estimate of a target position to insert the object into the insertion;
• controlling the robot to move to the estimated target position;
• taking a camera image using a camera after having controlled the robot to move to the estimated target position;
• feeding the camera image to the neural network trained by:
• controlling the robot to hold the object,
• bringing the robot into a first target position in which the object is inserted in the insertion,
• after bringing the robot into the first target position, for each first position of a plurality of first positions different from the first target position:
• controlling the robot to move away from the first target position to the first position,
• taking a first camera image by the camera, and
• labelling the first camera image by a movement vector to move back from the first position to the first target position and by an angle for correcting an orientation according to a difference between an insertion orientation and a current orientation,
• generating, for at least one of the camera images, an additional camera image by at least one of:
• colour jitter, conversion to grey scale, cropping, shifting and filtering of the camera image, and
• labelling the additional camera image with the movement vector with which the camera image is labelled, and
• training the neural network using the labelled first camera images and the labelled additional images;
• and
• controlling the robot to move according to a movement vector derived by the neural network from the camera image.