# US 10,124,489 B2 · Locating, separating, and picking boxes with a sensor-guided robot

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,124,489 B2
- Document ID: US10124489B2
- Status: Granted
- Kind: B2
- Application number: 15/419,862
- Filed: 2017-01-30
- Published: 2017-08-31
- Granted: 2018-11-13
- Earliest claimed: 2016-02-26
- Assignees: Kinema Systems Inc.
- Applicants: Kinema Systems Inc.
- Inventors: Sachin Chitta; David Hershberger; Karl Pauwels
- Web page: https://patentgrep.com/patent/US10124489B2

## CPC

- B25J 9/1687: {Assembly, peg and hole, palletising, straight line, weaving pattern movement}
- B65G 61/00: Use of pick-up or transfer devices or of manipulators for stacking or de-stacking articles not otherwise provided for
- G05B 2219/40006: Placing, palletize, un palletize, paper roll placing, box stacking
- G05B 2219/40607: Fixed camera to observe workspace, object, workpiece, global
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- B65G: TRANSPORT OR STORAGE DEVICES, e.g. CONVEYORS FOR LOADING OR TIPPING, SHOP CONVEYOR SYSTEMS OR PNEUMATIC TUBE CONVEYORS (packaging B65B; handling thin or filamentary materials, e.g. paper sheets or thread, B65H; cranes B66C; portable mobile or lifting or hauling appliances, e.g. hoists, B66D; devices for lifting or lowering goods for loading or unloading purposes, e.g. fork-lift trucks, B66F9/00; emptying bottles, jars, casks, barrels or similar containers, not otherwise provided for, B67C9/00; delivering or transferring liquids B67D; filling or discharging vessels for liquefied, solidified or compressed gases F17C; pipe-line systems for fluids F17D)
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

Techniques are described that enable robotic picking systems to locate and pick boxes from an unstructured pallet using computer vision and/or one or more “exploratory picks” to determine the sizes and locations of boxes on the pallet.

## Claims

### Claim 1

A computer-implemented method, comprising:
capturing a first image of a loaded pallet, the loaded pallet having a plurality of boxes stacked thereon;
generating a first representation of a surface of the loaded pallet using the first image;
positioning a robotic arm relative to a corner of the surface of the loaded pallet based on a minimum box size for the boxes on the loaded pallet;
moving a first box at the corner of the surface of the loaded pallet a programmable amount using the robotic arm;
capturing a second image of the loaded pallet including the first box as moved by the robotic arm;
generating a second representation of the surface of the loaded pallet using the second image;
releasing the first box with the robotic arm;
processing the first and second representations of the surface of the loaded pallet to generate a representation of a surface of the first box;
positioning the robotic arm relative to the first box using the representation of the surface of the first box; and
removing the first box from the loaded pallet using the robotic arm.

### Claim 2 (depends on 1)

The method of claim 1, wherein processing the first and second representations of the surface of the loaded pallet includes determining a difference between the first and second representations of the surface of the loaded pallet, and determining a size and a position of the surface of the first box using the difference.

### Claim 3 (depends on 1)

The method of claim 1, wherein the second image includes a representation of the robotic arm obscuring a portion of the surface of the first box, and wherein processing the first and second representations of the surface of the loaded pallet to generate a representation of the surface of the first box includes using data from both of the first and second representations of the surface of the loaded pallet to generate the representation of the surface of the first box.

### Claim 4 (depends on 1)

The method of claim 1, further comprising receiving input representing the minimum box size for the boxes on the loaded pallet, wherein positioning the robotic arm relative to the first box using the representation of the surface of the first box includes using the minimum box size.

### Claim 5 (depends on 1)

The method of claim 1, further comprising receiving input representing a maximum box size for the boxes on the loaded pallet, wherein removing the first box from the loaded pallet using the robotic arm includes (1) lifting the first box above a top surface of the loaded pallet by at least a height associated with the maximum box size, or (2) moving the first box away from a side surface of the loaded pallet by at least a distance associated with the maximum box size.

### Claim 6 (depends on 1)

The method of claim 1, wherein removing the first box from the loaded pallet using the robotic arm is accomplished without a priori knowledge of the number, sizes, orientations, and locations of the boxes on the loaded pallet.

### Claim 7 (depends on 1)

The method of claim 1, wherein the first and second images are captured using a camera, the method further comprising determining a location of the camera using a plurality of images of a fiducial held by the robotic arm in a corresponding plurality of positions.

### Claim 8 (depends on 1)

The method of claim 1, further comprising determining a location of the loaded pallet using an image of a fiducial on an empty pallet.

### Claim 9 (depends on 1)

The method of claim 1, further comprising unloading all of the boxes from the loaded pallet while performing the capturing, generating, lifting, capturing, generating, releasing, and processing for fewer than all of the boxes.

### Claim 10 (depends on 1)

The method of claim 1, wherein moving the first box comprises one of (1) lifting the first box the programmable amount relative to the loaded pallet using the robotic arm, (2) moving the first box laterally relative to the loaded pallet using the robotic arm, or (3) tilting the first box relative to the surface of the loaded pallet using the robotic arm.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
storing the representation of the surface of the first box;
capturing a third image of the loaded pallet using the image capture device, the third image not including the first box;
generating a third representation of the surface of the loaded pallet using the third image;
identifying one or more instances of a first box type corresponding to the first box by comparing the third representation of the surface of the loaded pallet and the stored representation of the surface of the first box, including determining a location and orientation of each of the instances of the first box type; and
picking each of the instances of the first box type using the corresponding location and orientation.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
storing the representation of the surface of the first box;
capturing an image of a different loaded pallet using the image capture device;
generating a representation of a surface of the different loaded pallet using the image;
identifying one or more instances of a first box type corresponding to the first box by comparing the representation of the surface of the different loaded pallet and the stored representation of the surface of the first box, including determining a location and orientation of each of the instances of the first box type; and
picking each of the instances of the first box type using the robotic arm and the corresponding location and orientation of each instance.

### Claim 13 (depends on 1)

The method of claim 1, wherein the first and second representations of the surface of the loaded pallet are three-dimensional representations.

### Claim 14 (depends on 1)

The method of claim 1, wherein moving the first box the programmable amount involves separating the first box from a top layer of the loaded pallet.

### Claim 15

A system, comprising a computing device, a robotic arm, a robotic control system, and an image capture device, the computing device being configured to:
capture a first image of a loaded pallet using the image capture device, the loaded pallet having a plurality of boxes stacked thereon;
generate a first representation of a surface of the loaded pallet using the first image;
position the robotic arm relative to a corner of the surface of the loaded pallet based on a minimum box size for the boxes on the loaded pallet;
move a first box at the corner of the surface of the loaded pallet a programmable amount using the robotic arm via the robotic control system;
capture a second image of the loaded pallet using the image capture device, the second image including the first box as moved by the robotic arm;
generate a second representation of the surface of the loaded pallet using the second image;
release the first box using the robotic arm via the robotic control system;
process the first and second representations of the surface of the loaded pallet to generate a representation of a surface of the first box;
position the robotic arm relative to the first box via the robotic control system using the representation of the surface of the first box; and
remove the first box from the loaded pallet using the robotic arm via the robotic control system.

### Claim 16 (depends on 15)

The system of claim 15, wherein the computing device is configured to process the first and second representations of the surface of the loaded pallet by determining a difference between the first and second representations of the surface of the loaded pallet, and determining a size and a position of the surface of the first box using the difference.

### Claim 17 (depends on 15)

The system of claim 15, wherein the second image includes a representation of the robotic arm obscuring a portion of the surface of the first box, and wherein the computing device is configured to process the first and second representations of the surface of the loaded pallet to generate a representation of the surface of the first box by using data from both of the first and second representations of the surface of the loaded pallet to generate the representation of the surface of the first box.

### Claim 18 (depends on 15)

The system of claim 15, wherein the computing device is further configured to receive input representing the minimum box size for the boxes on the loaded pallet, and wherein the computing device is configured to position the robotic arm relative to the first box using the representation of the surface of the first box by using the minimum box size.

### Claim 19 (depends on 15)

The system of claim 15, wherein the computing device is further configured to receive input representing a maximum box size for the boxes on the loaded pallet, and wherein the computing device is configured to remove the first box from the loaded pallet using the robotic arm by (1) lifting the first box above a top surface of the loaded pallet by at least a height associated with the maximum box size, or (2) moving the first box away from a side surface of the loaded pallet by at least a distance associated with the maximum box size.

### Claim 20 (depends on 15)

The system of claim 15, wherein the computing device is configured to remove the first box from the loaded pallet using the robotic arm without a priori knowledge of the number, sizes, orientations, and locations of the boxes on the loaded pallet.

### Claim 21 (depends on 15)

The system of claim 15, wherein the computing device is further configured to determine a location of the image capture device using a plurality of images of a fiducial held by the robotic arm in a corresponding plurality of positions.

### Claim 22 (depends on 15)

The system of claim 15, wherein the computing device is further configured to determine a location of the loaded pallet using an image of a fiducial on an empty pallet.

### Claim 23 (depends on 15)

The system of claim 15, wherein the computing device is further configured to unload all of the boxes from the loaded pallet using the robotic arm while performing the capturing, generating, lifting, capturing, generating, releasing, and processing for fewer than all of the boxes.

### Claim 24 (depends on 15)

The system of claim 15, wherein the computing device is further configured to move the first box by one of (1) lifting the first box the programmable amount relative to the loaded pallet using the robotic arm, (2) moving the first box laterally relative to the loaded pallet using the robotic arm, or (3) tilting the first box relative to the surface of the loaded pallet using the robotic arm.

### Claim 25 (depends on 15)

The system of claim 15, wherein the computing device is further configured to:
store the representation of the surface of the first box;
capture a third image of the loaded pallet using the image capture device, the third image not including the first box;
generate a third representation of the surface of the loaded pallet using the third image;
identify one or more instances of a first box type corresponding to the first box by comparing the third representation of the surface of the loaded pallet and the stored representation of the surface of the first box, including determining a location and orientation of each of the instances of the first box type; and
pick each of the instances of the first box type using the robotic arm and the corresponding location and orientation of each instance.

### Claim 26 (depends on 15)

The system of claim 15, wherein the computing device is further configured to:
store the representation of the surface of the first box;
capture an image of a different loaded pallet using the image capture device;
generate a representation of a surface of the different loaded pallet using the image;
identify one or more instances of a first box type corresponding to the first box by comparing the representation of the surface of the different loaded pallet and the stored representation of the surface of the first box, including determining a location and orientation of each of the instances of the first box type; and
pick each of the instances of the first box type using the robotic arm and the corresponding location and orientation of each instance.

### Claim 27 (depends on 15)

The system of claim 15, wherein the first and second representations of the surface of the loaded pallet are three-dimensional representations.

### Claim 28 (depends on 15)

The system of claim 15, wherein the computing device is configured to move the first box the programmable amount by separating the first box from a top layer of the loaded pallet.

### Claim 29

A computer program product, comprising one or more computer-readable media having computer program instructions stored therein, the computer program instructions being configured such that, when executed by one or more computing devices, the computer program instructions cause the one or more computing devices to:
capture a first image of a loaded pallet, the loaded pallet having a plurality of boxes stacked thereon;
generate a first representation of a surface of the loaded pallet using the first image;
position a robotic arm relative to a corner of the surface of the loaded pallet based on a minimum box size for the boxes on the loaded pallet;
move a first box at the corner of the surface of the loaded pallet a programmable amount using the robotic arm;
capture a second image of the loaded pallet including the first box as moved by the robotic arm;
generate a second representation of the surface of the loaded pallet using the second image;
release the first box using the robotic arm;
process the first and second representations of the surface of the loaded pallet to generate a representation of a surface of the first box;
position the robotic arm relative to the first box using the representation of the surface of the first box; and
remove the first box from the loaded pallet using the robotic arm.