# US 2020/0394747 A1 · Shading topography imaging for robotic unloading

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0394747 A1
- Document ID: US20200394747A1
- Status: Published application
- Kind: A1
- Application number: 16/898,903
- Filed: 2020-06-11
- Published: 2020-12-17
- Earliest claimed: 2019-06-12
- Assignees: Frito-Lay North America, Inc.; Graftek Imaging Inc.
- Applicants: Frito-Lay North America, Inc.; Graftek Imaging Inc.
- Inventors: Romik CHATTERJEE; Thomas M. STERRITT; Evan N. SANFORD; Clark A. TURPIN; Robert EASTLUND; Christopher KOCI; Nicholas CONRAD; Vincent IP; Sih Ying WU
- Web page: https://patentgrep.com/patent/US20200394747A1

## CPC

- G06T 1/0014: {Image feed-back for automatic industrial control, e.g. robot with camera (robots B25J19/023)}
- B25J 9/161: {Hardware, e.g. neural networks, fuzzy logic, interfaces, processor}
- B25J 9/1697: {Vision controlled systems}
- B25J 13/08: by means of sensing devices, e.g. viewing or touching devices
- B65G 67/24: Unloading land vehicles
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G05B 2219/37556: Camera detects fictive contour of workpiece, by reflection
- G05B 2219/40006: Placing, palletize, un palletize, paper roll placing, box stacking
- G05B 2219/40532: Ann for vision processing
- G05B 2219/40564: Recognize shape, contour of object, extract position and orientation
- G05B 2219/45056: Handling cases, boxes
- G06T 2207/10152: Varying illumination
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- B65G: TRANSPORT OR STORAGE DEVICES, e.g. CONVEYORS FOR LOADING OR TIPPING, SHOP CONVEYOR SYSTEMS OR PNEUMATIC TUBE CONVEYORS (packaging B65B; handling thin or filamentary materials, e.g. paper sheets or thread, B65H; cranes B66C; portable mobile or lifting or hauling appliances, e.g. hoists, B66D; devices for lifting or lowering goods for loading or unloading purposes, e.g. fork-lift trucks, B66F9/00; emptying bottles, jars, casks, barrels or similar containers, not otherwise provided for, B67C9/00; delivering or transferring liquids B67D; filling or discharging vessels for liquefied, solidified or compressed gases F17C; pipe-line systems for fluids F17D)
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

Vision systems for robotic assemblies for handling cargo, for example, unloading cargo from a trailer, can determine the position of cargo based on shading topography. Shading topography imaging can be performed by using light sources arranged at different positions relative to the image capture device(s).

## Claims

### Claim 1

A robotic assembly for handling cargo, the assembly comprising:
at least one robotic appendage for handling cargo; and
a vision system for acquiring information to assist operation of the at least one robotic appendage, the vision system comprising:
at least two light sources arranged spaced apart from each other;
at least one camera configured to capture images of cargo for handling by the robotic assembly; and
a vision control system for determining position of cargo, the vision control system including at least one processor for executing instructions stored on a memory to determine position of cargo based on a shading topography image comprising a blended composite of information of a plurality of images captured by the at least one camera.

### Claim 2 (depends on 1)

The robotic assembly of claim 1, wherein the shading topography image comprises a convolutional shading topography image comprising a convolution of the plurality of images captured by the at least one camera.

### Claim 3 (depends on 2)

The robotic assembly of claim 2, wherein the convolutional shading topography image is generated by a neural network.

### Claim 4 (depends on 2)

The robotic assembly of claim 2, wherein the convolution of the plurality of images comprises convolution of predefined channels of images captured by the at least one camera.

### Claim 5 (depends on 4)

The robotic assembly of claim 4, wherein one of the predefined channels comprises image data captured under illumination by at least one, but fewer than all, of the at least two light sources, and other image data captured with illumination by at least another one, but fewer than all, of the at least two light sources.

### Claim 6 (depends on 5)

The robotic assembly of claim 5, wherein the image data captured under illumination by at least one, but fewer than all, of the at least two light sources comprises image data captured under illumination by one of the at least two light sources having a first upper directional lighting trajectory and the other image data captured with illumination by at least another one, but fewer than all, of the at least two light sources comprises image data captured under illumination by another one of the at least two light sources having a first lower directional light trajectory.

### Claim 7 (depends on 6)

The robotic assembly of claim 6, wherein another one of the predefined channels comprises image data captured under illumination by at least one, but fewer than all, of the at least two light sources having a second upper directional light trajectory, different from the first upper directional light trajectory, and other image data captured with illumination by at least another one, but fewer than all, of the at least two light sources having a second lower directional light trajectory, different from the first lower directional light trajectory.

### Claim 8 (depends on 4)

The robotic assembly of claim 4, wherein at least one of the predefined channels comprises image data captured with illumination by greater than one of the at least two light sources.

### Claim 9 (depends on 4)

The robotic assembly of claim 4, wherein the image data captured with illumination by greater than one of the at least two light sources includes image data captured under illumination by first and second directional light trajectories for each of the at least two light sources.

### Claim 10 (depends on 1)

The robotic assembly of claim 1, wherein the at least one camera is arranged between two of the at least two light sources.

### Claim 11 (depends on 10)

The robotic assembly of claim 10, wherein the at least one camera is configured to capture at least one image of cargo under illumination by at least one of the at least two light sources and at least another image of cargo under illumination by another one of the at least two light sources.

### Claim 12 (depends on 11)

The robotic assembly of claim 11, wherein configuration of the at least one camera to capture the at least one image includes configuration to capture the at least one image under illumination by at least one of the at least two light sources without illumination by another of the at least two light sources, and configuration of the at least one camera to capture the at least another image includes configuration to capture the at least one another image under illumination by at least the another of the at least two light sources without illumination by at least one of the at least one of the at least two light sources.

### Claim 13 (depends on 1)

The robotic assembly of claim 1, wherein the at least one camera is coupled with a robotic unloading machine comprising the at least one robotic appendage.

### Claim 14 (depends on 13)

The robotic assembly of claim 13, wherein at least one of the at least two light sources is coupled with the robotic unloading machine.

### Claim 15 (depends on 1)

The robotic assembly of claim 1, wherein at least one of the at least two light sources includes a first light having a first directional lighting trajectory and a second light having a second light trajectory.

### Claim 16 (depends on 1)

The robotic assembly of claim 1, wherein the vision control system is adapted to conduct an imaging sequence including communicating with the at least one camera to capture one or more images of a wall of cargo under a predetermined illumination scheme of the at least two light sources.

### Claim 17 (depends on 16)

The robotic assembly of claim 16, wherein the predetermined illumination scheme includes one or more images having none of the at least two light sources illuminated, one or more images having all of the at least two light sources illuminated, and one or more images having fewer than all of the at least two light sources illuminated.

### Claim 18 (depends on 17)

The robotic assembly of claim 17, wherein the one or more images having fewer than all of the at least two light sources illuminated includes at least one image under illumination by only a first light of one of the at least two light sources having a first directional lighting trajectory.

### Claim 19 (depends on 18)

The robotic assembly of claim 18, wherein the one or more images having fewer than all of the at least two light sources illuminated includes at least one image under illumination by only a second light of the one of the at least two light sources having a second directional lighting trajectory, different from the first directional lighting trajectory.

### Claim 20 (depends on 1)

The robotic assembly of claim 1, wherein the shading topography image comprises an expression of absolute value of a gradient sum of intensity values of a number of images of cargo acquired by the at least one camera.

### Claim 21

A vision system of a robotic assembly for handling cargo, the vision system comprising:
at least two of light sources arranged spaced apart from each other;
at least one camera configured to capture images of cargo for handling by the robotic assembly; and
a vision control system for determining position of cargo, the vision control system including at least one processor for executing instructions stored on a memory to determine position of cargo based on a shading topography image comprising a blended composite of information of a plurality of images captured by the at least one camera.
