# US 2020/0130936 A1 · Identification and planning system and method for fulfillment of orders

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0130936 A1
- Document ID: US20200130936A1
- Status: Published application
- Kind: A1
- Application number: 16/555,301
- Filed: 2019-08-29
- Published: 2020-04-30
- Earliest claimed: 2018-10-25
- Assignees: Grey Orange Pte. Ltd.
- Applicants: Grey Orange Pte. Ltd.
- Inventors: Anirudh Singh SHEKHAWAT; Sameer NARKAR; Akash PATIL; Avilash KUMAR; Vaibhav TOLIA
- Web page: https://patentgrep.com/patent/US20200130936A1

## CPC

- B25J 9/1664: {characterised by motion, path, trajectory planning}
- B65G 1/1373: {for fulfilling orders in warehouses}
- G06Q 10/063: Operations research, analysis or management
- G06Q 10/08: Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
- G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders
- G06Q 10/08741: {by picking of items from inventory for fulfillment}
- B25J 9/1671: {characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems}
- B25J 9/1697: {Vision controlled systems}
- B65G 1/1378: {the orders being assembled on fixed commissioning areas remote from the storage areas}
- B65G 2203/0216: Codes or marks on the article
- G05B 2219/31286: Detect position of articles and equipment by receivers, identify objects by code
- G05B 2219/40005: Vision, analyse image at one station during manipulation at next station
- G06Q 10/08744: {by distributed inventory management}
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- B65G: TRANSPORT OR STORAGE DEVICES, e.g. CONVEYORS FOR LOADING OR TIPPING, SHOP CONVEYOR SYSTEMS OR PNEUMATIC TUBE CONVEYORS (packaging B65B; handling thin or filamentary materials, e.g. paper sheets or thread, B65H; cranes B66C; portable mobile or lifting or hauling appliances, e.g. hoists, B66D; devices for lifting or lowering goods for loading or unloading purposes, e.g. fork-lift trucks, B66F9/00; emptying bottles, jars, casks, barrels or similar containers, not otherwise provided for, B67C9/00; delivering or transferring liquids B67D; filling or discharging vessels for liquefied, solidified or compressed gases F17C; pipe-line systems for fluids F17D)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A identification and planning system includes a scanner assembly, a robotic manipulator, and control server. The control server detects a first plurality of inventory items in a first inventory storage system and determines a first set of position parameters for a first inventory item from the detected first plurality of inventory items, with respect to a position of one or more image sensors in the scanner assembly. The control server generates a plurality of pick-path plans for the first inventory item, where each pick-path plan corresponds to a transformation of the determined first set of position parameters to a second set of position parameters with respect to the robotic manipulator. The control server further selects a first pick-path plan from the plurality of pick-path plans and controls the robotic manipulator to pick the first inventory item from the first inventory storage system, based on the first pick-path plan.

## Claims

### Claim 1

An identification and planning system, comprising:
a scanner assembly that comprises one or more image sensors;
a robotic manipulator that comprises a robotic arm; and
a control server comprising control circuitry, wherein the control circuitry is configured to:
detect, by the one or more image sensors, a first plurality of inventory items in a first inventory storage system;
determine a first set of position parameters for a first inventory item from the detected first plurality of inventory items;
retrieve historical information associated with the first inventory item;
generate a plurality of pick-path plans for the first inventory item, wherein each pick-path plan of the plurality of pick-path plans corresponds to a transformation of the determined first set of position parameters to a second set of position parameters with respect to a spatial differentiation caused by a difference in position of the scanner assembly and the robotic manipulator;
select a first pick-path plan from the plurality of pick-path plans, wherein the first pick-path plan is selected before a pick operation is executed on the first inventory item, and the selection of the first pick-path plan is based on the retrieved historical information and a success of the pick operation in a test computer-simulation, the success of the pick operation occurs when the second set of position parameters corresponding to the selected first pick-path plan allow the robotic manipulator to reach and hold the first inventory item; and
in the pick operation, control, based on the first pick-path plan, the robotic arm to pick the first inventory item from the first inventory storage system.

### Claim 2 (depends on 1)

The identification and planning system of claim 1, wherein in the historical information for the first inventory item is updated after a successful pick up of the first inventory item.

### Claim 3 (depends on 1)

The identification and planning system of claim 1, wherein the historical information includes previous ordering information of the first inventory item.

### Claim 4 (depends on 1)

The identification and planning system of claim 1, wherein the scanner assembly is located at a position different than a position of the robotic manipulator such that scanner assembly is separated from the robotic manipulator.

### Claim 5 (depends on 1)

The identification and planning system according to claim 1, further comprising a first mobile robot, wherein the first inventory storage system is mounted on the first mobile robot that is configured to move the first inventory storage system from a first location to a second location in a warehouse based on instructions received from the control circuitry.

### Claim 6 (depends on 1)

The identification and planning system according to claim 1, wherein the first inventory storage system comprises a first plurality of storage bins, wherein each storage bin of the first plurality of storage bins is located at different spatial positions in the first inventory storage system, the first plurality of storage bins accommodates the first plurality of inventory items.

### Claim 7 (depends on 1)

The identification and planning system according to claim 1, wherein the control circuitry is further configured to execute, using the one or more image sensors, a scanning operation of the first plurality of inventory items in the first inventory storage system.

### Claim 8 (depends on 4)

The identification and planning system according to claim 4, wherein, in the scanning operation, the control circuitry is further configured to extract color information and depth information from a plurality of three dimensional (3D) images of the first plurality of inventory items, and
wherein the scanning operation further comprises the detection of the first plurality of inventory items and the determination of first set of position parameters.

### Claim 9 (depends on 1)

The identification and planning system according to claim 1, further comprising a second inventory storage system that comprises a second plurality of storage bins, wherein each storage bin of the second plurality of storage bins, present at different spatial positions in the second inventory storage system, accommodates a second plurality of inventory items.

### Claim 10 (depends on 6)

The identification and planning system according to claim 6, wherein the control circuitry is further configured to execute the pick operation for the first inventory item of the first inventory storage system, in parallel with a scanning operation on the second plurality of inventory items in one or more storage bins of the second inventory storage system.

### Claim 11 (depends on 1)

The identification and planning system according to claim 1, wherein the control circuitry is further configured to execute a queuing operation for the first inventory storage system and a second inventory storage system prior to an execution of the pick operation.

### Claim 12 (depends on 11)

The identification and planning system according to claim 11, further comprising at least one mobile robot,
wherein, in the queuing operation, the control circuitry is configured to control the at least one mobile robot to position the first inventory storage system and the second inventory storage system in a defined queue of a queuing station.

### Claim 13 (depends on 12)

The identification and planning system according to claim 12,
wherein the defined queue comprises grid regions, and
wherein the scanner assembly is positioned in front of a first grid region of the grid regions and the robotic manipulator is positioned in front of a second grid region of the grid regions.

### Claim 14 (depends on 13)

The identification and planning system according to claim 13,
wherein, in the queuing operation, the control circuitry is further configured to control the at least one mobile robot to move the first inventory storage system and the second inventory storage system through the grid regions of the defined queue, and
wherein the first inventory storage system and the second inventory storage system are moved for execution of the pick operation for the first inventory item in parallel with the scanning operation on the second plurality of inventory items in the second inventory storage system.

### Claim 15 (depends on 1)

The identification and planning system according to claim 1, wherein the control circuitry is further configured to:
estimate, based on a scan by one or more image sensors, three dimensional (3D) information that corresponds to different surface attributes of the first inventory item in the first inventory storage system of the first plurality of inventory items; and
estimate an orientation of the first inventory item in the first inventory storage system, based on the scan.

### Claim 16 (depends on 15)

The identification and planning system according to claim 15, wherein the control circuitry is further configured to apply a plurality of transform operations to the determined first set of position parameters to obtain the second set of position parameters, based on the estimated 3D surface information and the estimated orientation of the first inventory item.

### Claim 17 (depends on 16)

The identification and planning system according to claim 16, wherein the control circuitry is further configured to verify, in the test computer-simulation, the success of the pick operation on the first inventory item, based on execution of the pick operation corresponding to the application of each transform operation of the plurality of transform operations on the determined first set of position parameters.

### Claim 18 (depends on 1)

The identification and planning system according to claim 1, wherein the robotic manipulator further comprises a plurality of actuators in the robotic arm, wherein the plurality of actuators allow a movement of the robotic arm and the end effector within a defined number of degrees of freedom in a three dimensional (3D) space.

### Claim 19 (depends on 18)

The identification and planning system according to claim 18,
wherein the success of the pick operation corresponds to an event in the test computer-simulation, in which a simulation of the robotic manipulator picks the first inventory item using a simulated movement of the plurality of actuators, and
wherein the simulated movement is based on a defined number of degrees of freedom of the robotic arm and the second plurality of position parameters in the first pick-path plan.

### Claim 20 (depends on 1)

The identification and planning system according to claim 1, further comprising a memory configured to store a set of attributes of the first inventory item prior to a scanning operation and the pick operation,
wherein the control circuitry is further configured to estimate a set of grip parameters associated with the end effector of the robotic manipulator, based on the stored set of attributes of the first inventory item.

### Claim 21 (depends on 20)

The identification and planning system according to claim 20, wherein the control circuitry is further configured to verify, in the test computer-simulation, the success of the pick operation of the first inventory item by execution of the pick operation with a plurality of modifications in the estimated set of grip parameters.

### Claim 22 (depends on 21)

The identification and planning system according to claim 21, wherein the control circuitry is further configured to select the end effector from a set of end effectors, based on the stored set of attributes of the first inventory item.

### Claim 23 (depends on 1)

The identification and planning system according to claim 1, wherein each of the detected first plurality of inventory items in the first inventory storage system is associated with a unique tag that corresponds to a stock keeping unit (SKU).

### Claim 24 (depends on 23)

The identification and planning system according to claim 23, wherein the control circuitry is further configured to segregate, prior to a scanning operation and the pick operation, the first plurality of inventory items into a first set of inventory items and a second set of inventory items.

### Claim 25 (depends on 24)

The identification and planning system according to claim 24,
wherein the segregation of the first plurality of inventory items is based on the unique tag for each of the first plurality of inventory items, and
wherein the first set of inventory items correspond to inventory items that are auto pickable by the robotic manipulator and the second set of inventory items correspond to inventory items manually pickable by a warehouse operator.

### Claim 26 (depends on 1)

The identification and planning system according to claim 1, wherein the first set of position parameters are spatially different from the second set of position parameters, and wherein the spatial differentiation is caused by a difference in a spatial location of the scanner assembly and the robotic manipulator.

### Claim 27

An identification and planning system, comprising:
a control server comprising control circuitry, wherein the control circuitry is configured to:
detect a first plurality of inventory items in a first inventory storage system;
determine a first set of position parameters for a first inventory item from the detected first plurality of inventory items;
retrieve historical information associated with the first inventory item;
generate a plurality of pick-path plans for the first inventory item, wherein each pick-path plan of the plurality of pick-path plans corresponds to a transformation of the determined first set of position parameters to a second set of position parameters with respect to a spatial differentiation caused by a difference in position of the scanner assembly and a robotic manipulator;
select a first pick-path plan from the plurality of pick-path plans, wherein the first pick-path plan is selected before a pick operation is executed on the first inventory item, and the selection of the first pick-path plan is based on the retrieved historical information and a success of the pick operation in a test computer-simulation, the success of the pick operation occurs when the second set of position parameters corresponding to the selected first pick-path plan allow the robotic manipulator to reach and hold the first inventory item; and
in the pick operation, control, based on the selected first pick-path plan, a robotic arm to pick the first inventory item from the first inventory storage system.

### Claim 28

A method of an identification and planning system that includes control circuitry and one or more image sensors, the method comprising:
detecting, by the one or more image sensors, a first plurality of inventory items in a first inventory storage system;
determining, by the control circuitry, a first set of position parameters for a first inventory item from the detected first plurality of inventory items;
retrieving historical information associated with he first inventory item;
generating, by the control circuitry, a plurality of pick-path plans for the first inventory item, wherein each pick-path plan of the plurality of pick-path plans corresponds to a transformation of the determined first set of position parameters to a second set of position parameters with respect to a spatial differentiation caused by a difference in position of the scanner assembly and a robotic manipulator;
selecting, by the control circuitry, a first pick-path plan from the plurality of pick-path plans, wherein the first pick-path plan is selected before a pick operation is executed on the first inventory item, and the selection of the first pick-path plan is based on the retrieved historical information and a success of the pick operation in a test computer-simulation, the success of the pick operation occurs when the second set of position parameters corresponding to the selected first pick-path plan allow the robotic manipulator to reach and hold the first inventory item; and
in the pick operation, controlling, by the control circuitry and based on the selected first pick-path plan, a robotic arm and an end effector detachably attached to the robotic arm to pick the first inventory item from the first inventory storage system.
