Visual servoing of a robot

A system for handling an object, comprising:

• a robot head;
• a first vision sensor;
• an actuator configured to actuate said robot head;
• a device being connected to said first vision sensor and said robot head, said device comprising a processor and memory comprising instructions which cause the device to execute the following:
• obtaining, from said first vision sensor, an image of a scene comprising said object and a handle connected to said object, said image comprising 3D information;
• segmenting, by a trained segmentation neural network (NN), said image, according to a plurality of semantic components comprising at least a first semantic component relating to said object and a second semantic component relating to said handle;
• determining, based on said plurality of semantic components, handling data for handling said object, said handling data comprising a handling position being on said handle;
• computing, based on said handling data, a pose for said robot head, said pose comprising at least a robot head position for approaching said handle;
• and
• sending, to the actuator, actuation instructions for actuating said robot head toward said robot head position;
• wherein said first vision sensor is configured for:
• acquiring said image;
• sending the image to said device;
• wherein said actuator is configured for:
• receiving actuation instructions from said device;
• actuating said robot head in accordance with said actuation instructions,
• wherein the object belongs to a plurality of two or more objects comprised in said scene, and wherein the handle is shared by the plurality of objects being clustered objects, and
• wherein said system comprises a second vision sensor for performing a pre-scan of an environment for determining whether said handling is required, wherein said second vision sensor is mounted on a pre-scan portion of said system different from said robot head, and
• wherein the instructions further cause the device to execute a pre-scan which comprises:
• obtaining, by the second vision sensor, an environment image and performing semantic segmentation on the environment image by a trained pre-scan NN, yielding a pre-scan segmented 3D representation, and
• from this pre-scan segmented 3D representation, determining whether an object can be detected and, if yes, whether an approaching route can be found to handle the object for determining whether to carry out said (a) to (d).