Transparent object bin picking
AssigneeFanuc Corporation
InventorsTe Tang, Tetsuaki Kato
Life of this patent
Estimating the term…
Abstract
A system and method identifying an object, such as a transparent object, to be picked up by a robot from a bin of objects. The method includes obtaining a 2D red-green-blue (RGB) color image and a 2D depth map image of the objects using a 3D camera, where pixels in the depth map image are assigned a value identifying the distance from the camera to the objects. The method generates a segmentation image of the objects using a deep learning mask R-CNN (convolutional neural network) that performs an image segmentation process that extracts features from the RGB image and assigns a label to the pixels so that objects in the segmentation image have the same label. The method then identifies a location for picking up the object using the segmentation image and the depth map image.
Claims20
A method for picking up an object from a group of objects, said method comprising:
- obtaining a 2D red-green-blue (RGB) color image of the objects using a 3D camera;
- obtaining a 2D depth map image of the objects using the 3D camera, where pixels in the depth map image are assigned a value identifying the distance from the camera to the objects;
- generating a segmentation image of the objects by performing an image segmentation process that extracts features from the RGB image and assigns a label to pixels in the segmentation image so that the pixels in each object in the segmentation image have the same label and the pixels of different objects in the segmentation image have different labels including objects that have a same or similar shape, wherein obtaining a color image, obtaining a depth map image and generating a segmentation image are performed each time an object is picked up from the group of objects;
- and
- identifying a location for picking up the object using the segmentation image and the depth map image.
A method for picking up a transparent object from a group of transparent objects using a robot, said method comprising:
- obtaining a 2D red-green-blue (RGB) color image of the objects using a 3D camera;
- obtaining a 2D depth map image of the objects using the 3D camera, where pixels in the depth map image are assigned a value identifying the distance from the camera to the objects;
- generating a segmentation image of the objects by performing an image segmentation process using a deep learning mask R-CNN (convolutional neural network) that extracts features from the RGB image and assigns a label to pixels in the segmentation image so that each object in the segmentation image has the same label and the pixels of different objects in the segmentation image have different labels including objects that have a same or similar shape;
- and
- identifying a location for picking up the object using the segmentation image and the depth map image, wherein obtaining a color image, obtaining a depth map image, generating a segmentation image and identifying a location for picking up the object are performed each time an object is picked up from the group of objects by the robot.
A robot system for picking up an object from a group of objects using a robot, said system comprising:
- a 3D camera that provides a 2D red-green-blue (RGB) color image and a 2D depth map image of the objects;
- a deep learning mask R-CNN (convolutional neural network) that generates a segmentation image of the objects by performing an image segmentation process that extracts features from the RGB image and assigns a label to pixels in the segmentation image so that the pixels in each object in the segmentation image have the same label and the pixels of different objects in the segmentation image have different labels including objects that have a same or similar shape;
- and
- means for identifying a location for picking up the object using the segmentation image and the depth map image, wherein obtaining a color image, obtaining a depth map image, generating a segmentation image and identify a location for picking up the object are performed each time an object is picked up from the group of objects by the robot.
A method for picking up an object from a group of objects, said method comprising:
- obtaining a 2D red-green-blue (RGB) color image of the objects using a 3D camera;
- obtaining a 2D depth map image of the objects using the 3D camera, where pixels in the depth map image are assigned a value identifying the distance from the camera to the objects;
- generating a segmentation image of the objects by performing an image segmentation process that extracts features from the RGB image and assigns a label to pixels in the segmentation image so that each object in the segmentation image has the same label;
- and
- identifying a location for picking up the object using the segmentation image and the depth map image, wherein identifying a location for picking up the object includes identifying a center pixel of the object in the segmentation image, calculating x-y-z coordinates of the center pixel using the segmentation image and the depth map image, determining an x-y-z grasp position for picking up the object and identifying a grasp orientation for picking up the object that is defined by a ray direction from the camera to the center pixel.
Dependencies are inferred from the claim text.