# US 2024/0199333 A1 · Systems, apparatuses, and methods for warehouse operation and automatic order fulfillment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0199333 A1
- Document ID: US20240199333A1
- Status: Published application
- Kind: A1
- Application number: 18/591,223
- Filed: 2024-02-29
- Published: 2024-06-20
- Earliest claimed: 2021-08-30
- Assignees: Photoneo s. r. o.
- Applicants: Photoneo s. r. o.
- Inventors: Jan Zizka; Tomas Kovacovsky; Michal Maly
- Web page: https://patentgrep.com/patent/US20240199333A1

## CPC

- B65G 1/1373: {for fulfilling orders in warehouses}
- B65G 1/1378: {the orders being assembled on fixed commissioning areas remote from the storage areas}
- B25J 9/1666: {Avoiding collision or forbidden zones}
- B25J 9/1669: {characterised by special application, e.g. multi-arm co-operation, assembly, grasping}
- B25J 15/065: {provided with separating means for releasing the gripped object after suction}
- B65G 47/915: {provided with drive systems with rotary movements only}
- B66F 9/063: {Automatically guided}
- G05D 1/246: using environment maps, e.g. simultaneous localisation and mapping [SLAM]
- G05D 1/646: Following a predefined trajectory, e.g. a line marked on the floor or a flight path
- G05D 1/69: Coordinated control of the position or course of two or more vehicles
- G06V 10/82: using neural networks
- B65G 47/917: {control arrangements}
- B65G 2203/041: Camera
- G05D 2105/20: for transportation
- G05D 2107/70: Industrial sites, e.g. warehouses or factories
- B65G: TRANSPORT OR STORAGE DEVICES, e.g. CONVEYORS FOR LOADING OR TIPPING, SHOP CONVEYOR SYSTEMS OR PNEUMATIC TUBE CONVEYORS (packaging B65B; handling thin or filamentary materials, e.g. paper sheets or thread, B65H; cranes B66C; portable mobile or lifting or hauling appliances, e.g. hoists, B66D; devices for lifting or lowering goods for loading or unloading purposes, e.g. fork-lift trucks, B66F9/00; emptying bottles, jars, casks, barrels or similar containers, not otherwise provided for, B67C9/00; delivering or transferring liquids B67D; filling or discharging vessels for liquefied, solidified or compressed gases F17C; pipe-line systems for fluids F17D)
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- B66F: HOISTING, LIFTING, HAULING OR PUSHING, NOT OTHERWISE PROVIDED FOR, e.g. DEVICES WHICH APPLY A LIFTING OR PUSHING FORCE DIRECTLY TO THE SURFACE OF A LOAD ({invalid lifting devices A61G7/10; } mounting artificial islands on piles or like supports E02B17/00; scaffolding combined with lifting devices E04G1/22; lifting devices for sliding forms E04G11/24; lifting of buildings E04G23/06; shores or struts E04G25/00; lifts or other hoisting devices on ladders E06C7/12; props for mining E21D15/00)
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method for fulfillment of an order includes receiving, at a master server, a specification of the order including a set of items. The master server selects a runner apparatus and transmits a first command to that apparatus to retrieve a first box for presentation to a picker apparatus, where the first box has an item for fulfillment of the order. The runner apparatus retrieves the first box and presents it to the picker apparatus by moving near the picker apparatus, where the picker apparatus includes a base portion that is immovable, and holding the box for access by the picker apparatus. The master server instructs the picker apparatus to sort that item from the first box to a destination box. The picker apparatus picks the item out of the box via a robot arm of the picker apparatus, and moves it to the destination box.

## Claims

### Claim 1

A method for fulfillment of an order including a set of items disposed in a set of boxes, the boxes being disposed on a set of shelves, the method comprising:
a) receiving, at a master server, a specification of the order including the set of items;
b) generating, at the master server, a set of commands for fulfillment of the order; for each item of the set of items:
c) selecting a runner apparatus of a set of runner apparatuses as a first runner apparatus;
d) transmitting, by the master server to the first runner apparatus, a first command to retrieve a first box of the set of boxes for presentation to a picker apparatus, the first box having disposed therein a set of identical instances of that item;
e) retrieving, by the first runner apparatus and subject to the first command, the first box by:
moving to a location proximate to the first box;
via a lift of the first runner apparatus, moving a platform of the first runner apparatus to the same height as the first box;
drawing the first box off its associated shelf of the set of shelves and onto the platform;
f) presenting, by the first runner apparatus, the first box to the picker apparatus by:
moving to a location proximate to the picker apparatus, wherein the picker apparatus includes a base portion that is immovable; and
via the lift, moving the platform holding the first box to a predetermined height for access by the picker apparatus;
g) transmitting, by the master server the picker apparatus, a second command to sort that item from the first box to a destination box;
h) picking, by the picker apparatus and subject to the second command, that item by:
imaging at least a portion of an interior of the first box to identify and select an instance of that item from the set of items as a selected item;
identifying a pick point of the selected item; and
via a robot arm of the picker apparatus, engaging with the identified pick point of the selected item to pick the selected item out of the first box;
i) moving, by the picker apparatus, the selected item to the destination box by:
translocating the robot arm holding the selected item along a predetermined trajectory towards the destination box; and
releasing the selected item into the destination box.

### Claim 2 (depends on 1)

The method of claim 1, the drawing the first box off its associated shelf further including:
engaging a suction mechanism of platform with the first box to form a gripping hold between the platform and the first box; and
via a push-pull mechanism of the first runner apparatus, pulling the first box off its associated shelf and onto the platform.

### Claim 3 (depends on 1)

The method of claim 1, the imaging the portion of the interior of the first box to identify at least some of the set of items further including:
acquiring image and/or video data of the portion of the interior of the first box with an imaging device of the picker apparatus; and
applying a convolutional neural network to the image and/or video data to identify the at least some of the set of items.

### Claim 4 (depends on 1)

The method of claim 1, the identifying the pick point further comprising:
identifying a set of pick points on the selected item; and
selecting, a first pick point of the set of pick points as the identified pick point.

### Claim 5 (depends on 1)

The method of claim 1, wherein the robot arm includes a vacuum cup for picking the item, the picking the item further including, after engaging with the identified pick point:
detecting a pressure within the vacuum cup;
detect a flow rate within the vacuum cup; and
determining that the item was successfully picked when the ratio of the detected pressure to the detected flow rate exceeds a predetermined threshold.

### Claim 6 (depends on 1)

The method of claim 1, engaging with the identified pick point of the selected item further including:
applying suction pressure to a vacuum cup associated with the robot arm; and
engaging the vacuum cup with the identified pick point of the selected item to create a vacuum-sealed grip between the vacuum cup and the selected item.

### Claim 7 (depends on 1)

The method of claim 1, the picking, by the picker apparatus and subject to the second command, that item further including:
j) generating, in a randomized manner, a spatial trajectory for movement of the robot arm untaken to pick that item as the selected item and to move the selected item to the destination box;
k) analyzing that spatial trajectory to identify any collisions of the robot arm with its environment;
l) if no collisions are identified at step (k), deeming that spatial trajectory as the predetermined trajectory; and
m) if a collision is identified, repeating steps (j)-(m).

### Claim 8 (depends on 7)

The method of claim 7, step (j) further including generating a polyhedral representation of the robot arm and of its environment, and step (k) further including identifying a collision between the robot arm and its environment when the polyhedral representation of the robot arm spatially overlaps with the polyhedral representation of its environment at any point on that trajectory.

### Claim 9 (depends on 1)

The method of claim 1, further comprising executing steps (c)-(i) for two or more items of the set of items in an overlapping manner.

### Claim 10 (depends on 1)

The method of claim 1, wherein the order is a first order, the picker apparatus is first picker apparatus, and the set of runner apparatuses is a first set of runner apparatuses, such that the first picker apparatus and the first set of runner apparatuses constitute a first segment, further comprising:
receiving, at the master server, a specification of a second order including the set of items;
repeating steps (b)-(i) for the second order with a second segment including a second picker apparatus and a second set of runner apparatuses.

### Claim 11

A system for fulfillment of an order including a set of items disposed in a set of boxes, the boxes being disposed on a set of shelves, the system comprising:
a set of runner apparatuses, each runner apparatus of the set of runner apparatuses including:
a platform for holding a box; and
a lift coupled to the platform for vertical translocation of the platform;
a picker apparatus including:
a base portion that is immovable; and
a robot arm coupled to the base portion to removably engage with items to be picked; and
a server communicably coupled to the set of runner apparatuses and to the picker apparatus, the server configured to:
receive a specification of the order including the set of items;
generate a set of commands for fulfillment of the order;
for each item of the set of items:
select a runner apparatus from the set of runner apparatuses as a first runner apparatus;
transmit, to the first runner apparatus, a first command to retrieve a first box of the set of boxes for presentation to the picker apparatus, the first box having disposed therein a set of identical instances of that item; and
transmit, to the picker apparatus, a second command to sort that item from the first box to a destination box,
wherein the first runner apparatus is configured to:
retrieve by the first runner apparatus and subject to the first command, the first box by:
moving to a location proximate to the first box;
via the lift of the first runner apparatus, moving the platform of the first runner apparatus to the same height as the first box; and
drawing the first box off its associated shelf of the set of shelves and onto the platform; and
present the first box to the picker apparatus by:
moving to a location proximate to the picker apparatus; and
via the lift of the first runner apparatus, moving the platform holding the first box to a predetermined height for access by the picker apparatus,
wherein the picker apparatus is configured to:
pick, subject to the second command, that item by:
imaging at least a portion of an interior of the first box to identify and select an instance of that item from the set of items as a selected item;
identifying a pick point of the selected item; and
via the robot arm of the picker apparatus, engaging with the identified pick point of the selected item to pick the selected item out of the first box; and
move the selected item to the destination box by:
translocating the robot arm holding the selected item along a predetermined trajectory towards the destination box; and
releasing the selected item into the destination box.

### Claim 12 (depends on 10)

The system of claim 10, wherein the picker apparatus further includes an imaging device, and wherein the picker apparatus is further configured to image the portion of the interior of the first box to identify at least some of the set of items by:
acquiring image and/or video data of the portion of the interior of the first box with the imaging device of the picker apparatus; and
applying a convolutional neural network to the image and/or video data to identify the at least some of the set of items.

### Claim 13 (depends on 10)

The system of claim 10, wherein the picker processor is further configured to identify the pick point by:
identifying a set of pick points on the selected item; and
selecting, a first pick point of the set of pick points as the identified pick point.

### Claim 14 (depends on 10)

The system of claim 10, wherein the picker apparatus is further configured to pick that item by:
generating, in a randomized manner, a spatial trajectory for movement of the robot arm taken to pick that item as the selected item and to move the selected item to the destination box;
analyzing that spatial trajectory to identify any collisions of the robot arm with its environment;
if no collisions are identified, deeming that spatial trajectory as the predetermined trajectory; and
if a collision is identified, repeating said generating and analyzing steps.

### Claim 15 (depends on 14)

The system of claim 14, wherein the picker apparatus is further configured to generate the spatial trajectory by generating a polyhedral representation of the robot arm and of its environment,
and wherein the picker apparatus is further configured to analyze the spatial trajectory by identifying a collision between the robot arm and its environment when the polyhedral representation of the robot arm spatially overlaps with the polyhedral representation of its environment at any point on that trajectory.

### Claim 16 (depends on 10)

The system of claim 10, wherein the picker apparatus further includes:
a vacuum cup coupled to the robot arm, to removably engage with items to be picked;
a vacuum pump coupled to the vacuum cup, to apply a suction force via the vacuum cup to items to be picked; and
a pressure meter coupled to the vacuum cup, to detect a pressure within the vacuum cup.

### Claim 17 (depends on 16)

The system of claim 16, wherein the picker apparatus is further configured to transmit an indication of the pressure within the vacuum cup to the server, and wherein the server is further configured determine that the selected item has been picked from the first box by the robot arm.

### Claim 18 (depends on 16)

The system of claim 16, wherein the picker apparatus further includes a flow meter coupled to the vacuum cup, to detect a flow rate within the vacuum cup,
wherein the picker apparatus is further configured to, when the ratio of the detected pressure to the detected flow rate exceeds a predetermined threshold, transmit an indication that the selected item has been picked from the first box by the robot arm.

### Claim 19 (depends on 10)

The system of claim 10, wherein the first runner apparatus further includes one or more suction cups for engaging with a box to be manipulated, and wherein the first runner apparatus is configured to draw the first box onto the platform by:
coupling the one or more suction cups with the first box via a suction force applied to the one or more suction cups; and
moving the one or more suction cups across the platform to pull the first box coupled thereto onto the platform.

### Claim 20 (depends on 10)

The system of claim 10, wherein each runner apparatus of the set of runner apparatuses includes a set of motion sensors to monitor change in position of that runner apparatus, wherein each runner apparatus of the set of runner apparatuses is further configured to:
maintain and/or update a spatial map of their surroundings using simultaneous localization and mapping (SLAM); and
based on its set of motion sensors, determine its location in real-time on the spatial map,
and wherein the first runner apparatus is further configured to move to a location proximate to the picker apparatus based on its corresponding spatial map and continuous monitoring of its corresponding set of motion sensors.
