Object bin picking with rotation compensation

A method for picking up a transparent object from a group of transparent objects using a robot, said method comprising:

• obtaining a 2D red-green-blue (RGB) color image of the objects using a 3D camera;
• obtaining a 2D depth map image of the objects using the 3D camera, where pixels in the depth map image are assigned a value identifying the distance from the camera to the objects;
• generating a segmentation image of the objects by performing an image segmentation process using a deep learning neural network that extracts features from the RGB image, assigns a label to pixels in the segmentation image so that each object in the segmentation image has the same label and determines an orientation of the objects;
• identifying a location for picking up the object using the segmentation image and the depth map image;
• and
• rotating the object using the orientation of the object in the segmentation image, wherein obtaining a color image, obtaining a depth map image, generating a segmentation image, identifying a location for picking up the object and rotating the object are performed each time an object is picked up from the group of objects by the robot.

A robot system for picking up an object from a group of objects using a robot, said system comprising:

• a camera that provides an image of the objects;
• a neural network that generates a segmentation image of the objects by performing an image segmentation process that extracts features from the image and assigns a label to pixels in the segmentation image so that each object in the segmentation image has the same label, wherein the neural network provides a plurality of bounding boxes having the same orientation, aligns the bounding boxes to the extracted features using a sliding window template, provides a bounding box image that includes bounding boxes surrounding the objects, determines the probability that an object exists in each bounding box and identifies a center pixel of each object in the bounding boxes;
• means for identifying a location for picking up the object using the segmentation image;
• and
• means for rotating the object using the orientation of the object in the segmentation image, wherein obtaining an image, generating a segmentation image, identify a location for picking up the object and rotating the object are performed each time an object is picked up from the group of objects by the robot.

A method for picking up an object from a group of objects, said method comprising:

• obtaining an image of the objects using a camera;
• generating a segmentation image of the objects by performing an image segmentation process that includes using neural networks to extract features from the image to provide an extracted features image, identifying x and y coordinates of pixels in the extracted features image and identifying an orientation of the objects in the extracted features image, wherein generating a segmentation image includes identifying a center pixel of each object, providing a plurality of bounding boxes, aligning the bounding boxes to the extracted features and providing a bounding box image that includes bounding boxes surrounding the objects;
• identifying a location for picking up the object using the segmentation image, wherein identifying a location for picking up the object includes identifying the center pixel of one of the objects in the segmentation image and calculating x-y-z coordinates of the center pixel using the location of that center pixel and a depth map image;
• and
• rotating the object using the orientation of the object in the segmentation image, wherein the objects are transparent.