# US 12,112,499 B2 · Algorithm for mix-size depalletizing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,499 B2
- Document ID: US12112499B2
- Status: Granted
- Kind: B2
- Application number: 17/456,977
- Filed: 2021-11-30
- Published: 2023-06-01
- Granted: 2024-10-08
- Earliest claimed: 2021-11-30
- Assignees: Fanuc Corporation
- Applicants: Fanuc Corporation
- Inventors: Te Tang; Tetsuaki Kato
- Web page: https://patentgrep.com/patent/US12112499B2

## CPC

- G06T 7/73: using feature-based methods
- B25J 9/1687: {Assembly, peg and hole, palletising, straight line, weaving pattern movement}
- B25J 9/1697: {Vision controlled systems}
- G06T 7/11: Region-based segmentation
- G06T 7/12: Edge-based segmentation
- G06T 7/155: involving morphological operators
- G06T 7/187: involving region growing; involving region merging; involving connected component labelling
- G06T 7/194: involving foreground-background segmentation
- G06T 7/33: using feature-based methods
- G06T 7/593: from stereo images
- G06V 10/34: Smoothing or thinning of the pattern; Morphological operations; Skeletonisation
- G06V 10/82: using neural networks
- G06V 20/64: Three-dimensional [3D] objects
- G05B 2219/40006: Placing, palletize, un palletize, paper roll placing, box stacking
- G05B 2219/40607: Fixed camera to observe workspace, object, workpiece, global
- G06T 2207/10012: Stereo images
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20036: Morphological image processing
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A system and method for identifying a box to be picked up by a robot from a stack of boxes. The method includes obtaining a 2D red-green-blue (RGB) color image of the boxes and a 2D depth map image of the boxes using a 3D camera. The method employs an image segmentation process that uses a simplified mask R-CNN executable by a central processing unit (CPU) to predict which pixels in the RGB image are associated with each box, where the pixels associated with each box are assigned a unique label that combine to define a mask for the box. The method then identifies a location for picking up the box using the segmentation image.

## Claims

### Claim 1

A method for picking up an object box from a group of object boxes, said method comprising:
obtaining a 2D red-green-blue (RGB) color image of the boxes using a 3D camera;
obtaining a 2D depth map image of the boxes using the 3D camera, where pixels in the depth map image are assigned a value identifying the distance from the camera to the boxes;
generating a segmentation mask image of the boxes by performing an image segmentation process in a neural network, said image segmentation process extracting features from the RGB image and using the extracted features to predict which pixels in the RGB image are associated with each box, where the pixels associated with each box are assigned a unique label that combine to define a mask for the box;
selecting one of the masks in the segmentation mask image;
identifying and removing pixels in the selected mask that may be associated with a lower object box than the object box associated with the selected mask to provide a refined mask;
generating a lower boundary mask and an upper boundary mask from the refined mask using a morphological process;
generating an expanded mask from the lower boundary mask and the upper boundary mask using a refinement process, wherein generating an expanded mask includes generating an initial image that only includes the region of the RBG image defined by the upper boundary mask where each pixel's value in the initial image is a light intensity, then generating a gradient image from the initial image where each pixel intensity is replaced with a gradient intensity based on neighbor pixels, and then dilating the pixel intensities to identify peaks in the gradient image; and
identifying a location for picking up the box using the expanded mask.

### Claim 2 (depends on 1)

The method according to claim 1 wherein the neural network is a simplified mask R-CNN (convolutional neural network).

### Claim 3 (depends on 1)

The method according to claim 1 wherein generating a segmentation mask image includes providing a plurality of bounding boxes, aligning the bounding boxes to the extracted features and providing a bounding box image that includes bounding boxes surrounding the object boxes.

### Claim 4 (depends on 3)

The method according to claim 3 wherein generating a segmentation mask image includes determining the probability that an object box exists in each bounding box.

### Claim 5 (depends on 3)

The method according to claim 3 wherein generating a segmentation mask image includes removing pixels from each bounding box in the bounding box image that are not associated with an object box.

### Claim 6 (depends on 1)

The method according to claim 1 wherein identifying and removing pixels in the selected mask includes using the depth map image.

### Claim 7 (depends on 1)

The method according to claim 1 wherein generating a lower boundary mask and an upper boundary mask includes performing a dilation process where a predetermined number of pixels are added to an outer boundary of the refined mask and an erosion process where a predetermined number of pixels are removed from the outer boundary of the refined mask.

### Claim 8 (depends on 1)

The method according to claim 1 further comprising determining an orientation angle (θ) of the expanded mask by fitting a minimum size bounding box around the expanded mask.

### Claim 9 (depends on 8)

The method according to claim 8 wherein fitting a minimum size bounding box around the expanded mask includes using an iterative minimum rectangle search to obtain a smallest size bounding box that surrounds the expanded mask.

### Claim 10 (depends on 1)

The method according to claim 1 wherein identifying a location for picking up the object box includes identifying a center pixel of final boundary mask and calculating x-y-z coordinates of the center pixel using the location of the center pixel and the depth map image.

### Claim 11 (depends on 10)

The method according to claim 10 wherein identifying a location for picking up the object box includes determining a grasp pose for picking up the object box using the x-y-z coordinates and a pitch, yaw and roll position.

### Claim 12 (depends on 11)

The method according to claim 11 wherein the object box is picked up by a robot and the grasp pose determines the pose of a robot end-effector.

### Claim 13

A method for selecting an object from a group of objects, said method comprising:
obtaining a 2D red-green-blue (RGB) color image of the objects using a 3D camera;
generating a segmentation mask image of the objects by performing an image segmentation process in a neural network, said image segmentation process extracting features from the RGB image and using the extracted features to predict which pixels in the RGB image are associated with each object, where the pixels associated with each object are assigned a unique label that combine to define a mask for the object;
selecting one of the masks in the segmentation mask image;
identifying and removing pixels in the selected mask that may be associated with a lower object than the object associated with the selected mask to provide a refined mask;
generating a lower boundary mask and an upper boundary mask from the refined mask using a morphological process;
generating an expanded mask from the lower boundary mask and the upper boundary mask using a refinement process, wherein generating an expanded mask includes generating an initial image that only includes the region of the RBG image defined by the upper boundary mask where each pixel's value in the initial image is a light intensity, then generating a gradient image from the initial image where each pixel intensity is replaced with a gradient intensity based on neighbor pixels, and then dilating the pixel intensities to identify peaks in the gradient image; and
identifying a location of the object using the expanded mask.

### Claim 14 (depends on 13)

The method according to claim 13 wherein the neural network is a simplified mask R-CNN (convolutional neural network).

### Claim 15 (depends on 13)

The method according to claim 13 wherein generating a segmentation mask image includes providing a plurality of bounding boxes, aligning the bounding boxes to the extracted features and providing a bounding box image that includes bounding boxes surrounding the objects.

### Claim 16 (depends on 15)

The method according to claim 15 wherein generating a segmentation mask image includes determining the probability that an object exists in each bounding box.

### Claim 17 (depends on 15)

The method according to claim 15 wherein generating a segmentation mask image includes removing pixels from each bounding box in the bounding box image that are not associated with an object.

### Claim 18 (depends on 13)

The method according to claim 13 wherein identifying and removing pixels in the selected mask includes using a depth map image.

### Claim 19 (depends on 13)

The method according to claim 13 wherein generating a lower boundary mask and an upper boundary mask includes performing a dilation process where a predetermined number of pixels are added to an outer boundary of the refined mask and an erosion process where a predetermined number of pixels are removed from the outer boundary of the refined mask.

### Claim 20 (depends on 13)

The method according to claim 13 further comprising determining an orientation angle (θ) of the expanded mask by fitting a minimum size bounding box around the expanded mask.

### Claim 21 (depends on 20)

The method according to claim 20 wherein fitting a minimum size bounding box around the expanded mask includes using an iterative minimum rectangle search to obtain a smallest size bounding box that surrounds the expanded mask.

### Claim 22

A system for picking up an object box from a group of object boxes, said system comprising:
means for obtaining a 2D red-green-blue (RGB) color image of the boxes using a 3D camera;
means for obtaining a 2D depth map image of the boxes using the 3D camera, where pixels in the depth map image are assigned a value identifying the distance from the camera to the boxes;
means for generating a segmentation mask image of the boxes by performing an image segmentation process in a neural network, said image segmentation process extracting features from the RGB image and using the extracted features to predict which pixels in the RGB image are associated with each box, where the pixels associated with each box are assigned a unique label that combine to define a mask for the box;
means for selecting one of the masks in the segmentation mask image;
means for identifying and removing pixels in the selected mask that may be associated with a lower object box than the object box associated with the selected mask to provide a refined mask;
means for generating a lower boundary mask and an upper boundary mask from the refined mask using a morphological process;
means for generating an expanded mask from the lower boundary mask and the upper boundary mask using a refinement process, wherein the means for generating an expanded mask generates an initial image that only includes the region of the RBG image defined by the upper boundary mask where each pixel's value in the initial image is a light intensity, then generating a gradient image from the initial image where each pixel intensity is replaced with a gradient intensity based on neighbor pixels, and then dilating the pixel intensities to identify peaks in the gradient image; and
means for identifying a location for picking up the box using the expanded mask.

### Claim 23 (depends on 1)

The method according to claim 1 wherein the lower boundary mask is smaller and includes less pixels than the refined mask and the upper boundary mask is larger and includes more pixels than the refined mask.

### Claim 24 (depends on 13)

The method according to claim 13 wherein the lower boundary mask is smaller and includes less pixels than the refined mask and the upper boundary mask is larger and includes more pixels than the refined mask.

### Claim 25 (depends on 22)

The system according to claim 22 wherein the lower boundary mask is smaller and includes less pixels than the refined mask and the upper boundary mask is larger and includes more pixels than the refined mask.