Back to results
US 12,112,499 B2Granted
Algorithm for mix-size depalletizing
AssigneeFanuc Corporation
InventorsTe Tang, Tetsuaki Kato
Life of this patent
Estimating the term…
Abstract
A system and method for identifying a box to be picked up by a robot from a stack of boxes. The method includes obtaining a 2D red-green-blue (RGB) color image of the boxes and a 2D depth map image of the boxes using a 3D camera. The method employs an image segmentation process that uses a simplified mask R-CNN executable by a central processing unit (CPU) to predict which pixels in the RGB image are associated with each box, where the pixels associated with each box are assigned a unique label that combine to define a mask for the box. The method then identifies a location for picking up the box using the segmentation image.
Claims25
A method for picking up an object box from a group of object boxes, said method comprising:
- obtaining a 2D red-green-blue (RGB) color image of the boxes using a 3D camera;
- obtaining a 2D depth map image of the boxes using the 3D camera, where pixels in the depth map image are assigned a value identifying the distance from the camera to the boxes;
- generating a segmentation mask image of the boxes by performing an image segmentation process in a neural network, said image segmentation process extracting features from the RGB image and using the extracted features to predict which pixels in the RGB image are associated with each box, where the pixels associated with each box are assigned a unique label that combine to define a mask for the box;
- selecting one of the masks in the segmentation mask image;
- identifying and removing pixels in the selected mask that may be associated with a lower object box than the object box associated with the selected mask to provide a refined mask;
- generating a lower boundary mask and an upper boundary mask from the refined mask using a morphological process;
- generating an expanded mask from the lower boundary mask and the upper boundary mask using a refinement process, wherein generating an expanded mask includes generating an initial image that only includes the region of the RBG image defined by the upper boundary mask where each pixel's value in the initial image is a light intensity, then generating a gradient image from the initial image where each pixel intensity is replaced with a gradient intensity based on neighbor pixels, and then dilating the pixel intensities to identify peaks in the gradient image;
- and
- identifying a location for picking up the box using the expanded mask.
6 more elements
A method for selecting an object from a group of objects, said method comprising:
- obtaining a 2D red-green-blue (RGB) color image of the objects using a 3D camera;
- generating a segmentation mask image of the objects by performing an image segmentation process in a neural network, said image segmentation process extracting features from the RGB image and using the extracted features to predict which pixels in the RGB image are associated with each object, where the pixels associated with each object are assigned a unique label that combine to define a mask for the object;
- selecting one of the masks in the segmentation mask image;
- identifying and removing pixels in the selected mask that may be associated with a lower object than the object associated with the selected mask to provide a refined mask;
- generating a lower boundary mask and an upper boundary mask from the refined mask using a morphological process;
- generating an expanded mask from the lower boundary mask and the upper boundary mask using a refinement process, wherein generating an expanded mask includes generating an initial image that only includes the region of the RBG image defined by the upper boundary mask where each pixel's value in the initial image is a light intensity, then generating a gradient image from the initial image where each pixel intensity is replaced with a gradient intensity based on neighbor pixels, and then dilating the pixel intensities to identify peaks in the gradient image;
- and
- identifying a location of the object using the expanded mask.
5 more elements
A system for picking up an object box from a group of object boxes, said system comprising:
- means for obtaining a 2D red-green-blue (RGB) color image of the boxes using a 3D camera;
- means for obtaining a 2D depth map image of the boxes using the 3D camera, where pixels in the depth map image are assigned a value identifying the distance from the camera to the boxes;
- means for generating a segmentation mask image of the boxes by performing an image segmentation process in a neural network, said image segmentation process extracting features from the RGB image and using the extracted features to predict which pixels in the RGB image are associated with each box, where the pixels associated with each box are assigned a unique label that combine to define a mask for the box;
- means for selecting one of the masks in the segmentation mask image;
- means for identifying and removing pixels in the selected mask that may be associated with a lower object box than the object box associated with the selected mask to provide a refined mask;
- means for generating a lower boundary mask and an upper boundary mask from the refined mask using a morphological process;
- means for generating an expanded mask from the lower boundary mask and the upper boundary mask using a refinement process, wherein the means for generating an expanded mask generates an initial image that only includes the region of the RBG image defined by the upper boundary mask where each pixel's value in the initial image is a light intensity, then generating a gradient image from the initial image where each pixel intensity is replaced with a gradient intensity based on neighbor pixels, and then dilating the pixel intensities to identify peaks in the gradient image;
- and
- means for identifying a location for picking up the box using the expanded mask.
6 more elements
Dependencies are inferred from the claim text.