# US 11,077,554 B2 · Controller and control method for robotic system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,077,554 B2
- Document ID: US11077554B2
- Status: Granted
- Kind: B2
- Application number: 16/886,734
- Filed: 2020-05-28
- Published: 2020-12-03
- Granted: 2021-08-03
- Earliest claimed: 2019-05-31
- Assignees: MUJIN, Inc.
- Applicants: MUJIN, Inc.
- Inventors: Isamu Arase; Satoshi Tendo; Rosen Nikolaev Diankov
- Web page: https://patentgrep.com/patent/US11077554B2

## CPC

- B25J 9/1664: {characterised by motion, path, trajectory planning}
- B25J 9/1687: {Assembly, peg and hole, palletising, straight line, weaving pattern movement}
- B65G 61/00: Use of pick-up or transfer devices or of manipulators for stacking or de-stacking articles not otherwise provided for
- B65G 65/00: Loading or unloading (of vehicles B65G67/00)
- G05B 19/41895: {using automatic guided vehicles [AGV] (control of position or course of AGV's G05D1/00)}
- G05B 2219/40006: Placing, palletize, un palletize, paper roll placing, box stacking
- Y02P 90/02: Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
- Y02P 90/60: Electric or hybrid propulsion means for production processes
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- B65G: TRANSPORT OR STORAGE DEVICES, e.g. CONVEYORS FOR LOADING OR TIPPING, SHOP CONVEYOR SYSTEMS OR PNEUMATIC TUBE CONVEYORS (packaging B65B; handling thin or filamentary materials, e.g. paper sheets or thread, B65H; cranes B66C; portable mobile or lifting or hauling appliances, e.g. hoists, B66D; devices for lifting or lowering goods for loading or unloading purposes, e.g. fork-lift trucks, B66F9/00; emptying bottles, jars, casks, barrels or similar containers, not otherwise provided for, B67C9/00; delivering or transferring liquids B67D; filling or discharging vessels for liquefied, solidified or compressed gases F17C; pipe-line systems for fluids F17D)
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- Y02P: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS

## Abstract

A controller for a robotic system is provided. The controller may derive first data including information about operation objects and information about pallet before loading and/or unloading of the operation object on/from the pallet. The controller may derive a control sequence for loading and/or unloading the operation object on/from the pallet based on the first data before execution of the corresponding loading/unloading task. Based on the control sequence, the controller may instruct a robot to execute the task of loading and/or unloading the operation object on/from the pallet.

## Claims

### Claim 1

A controller for controlling an operation robot to manipulate operation objects to load the operation object on a pallet and/or unload the operation objects from the pallet, the controller comprising:
a data deriving unit for deriving:
first data including information about an operation object and information about the pallet before loading of the operation object on the pallet and/or before unloading of the operation object from the pallet, and
second data including information about the operation object and information about the pallet after loading of the operation object on the pallet and/or after unloading of the operation object from the pallet;
a data storage unit for storing the first data and the second data; and
a robot control unit for:
creating or deriving, at least in part, a control sequence for loading of the operation object on the pallet and/or unloading of the operation object from the pallet based on the first data before loading of the operation object on the pallet and/or before unloading of the operation object from the pallet, and
instructing the operation robot to, using an end effector of the robotic and based on the control sequence, execute a task of loading the operation object on the pallet and/or unloading the operation object from the pallet,
wherein:
the robot control unit creates or derives the control sequence based on the first data and/or the second data such that:
at least two operation objects having different stock keeping units from one another are loaded on the pallet, and
a plurality of operation objects having same or similar stock keeping units are placed on the pallet in a concentrated manner, and
before loading of the operation object on the pallet and/or unloading of the operation object from the pallet are/is actually finished, the robot control unit determines that the loading of the operation object on the pallet and/or the unloading of the operation object from the pallet are/is completed.

### Claim 2 (depends on 1)

The controller of claim 1, wherein the robot control unit:
calculates, based on the second data, an index indicating handling property for unloading of the operation object from the pallet, and
based on the index, creates or derives the control sequence including identification of the operation object and the pallet loaded with the operation object.

### Claim 3 (depends on 1)

The controller of claim 1, wherein the data storage unit stores positional information about a plurality of the operation objects on the pallet as two-dimensional information or three-dimensional information for each operation object of the plurality or for each pallet.

### Claim 4 (depends on 1)

The controller of claim 1, wherein:
the operation object includes an identification code or tag attached to a side of the operation object,
the robot control unit creates or derives the control sequence for loading the operation object on the pallet so as to allow a sensor to image or read the identification code or tag after the operation object is loaded on the pallet, and
the sensor is configured for imaging of the operation object or of the pallet loaded with the operation object.

### Claim 5 (depends on 1)

The controller of claim 1, wherein the data deriving unit derives measured values or estimated values of the information about the operation object and/or of the information about the pallet.

### Claim 6 (depends on 1)

The controller of claim 1, wherein:
the controller identifies (i) a robotic system that includes the operation robot and (ii) a region related to the robotic system, and calculates tasks based on the control sequence; and
the tasks include a task related to transportation of the pallet and operating of the operation object by the robotic system and a plurality of tasks to be executed by the robotic system across adjacent and/or overlapping regions.

### Claim 7

A tangible, non-transitory computer readable medium having processor instructions stored thereon that, when executed by one or more processors of a computer, cause the computer to perform a method for controlling an operation robot for manipulating operation objects to load the operation objects on a pallet and/or unload the operation objects from the pallet, and wherein the method comprises:
deriving first data including information about an operation object and information about the pallet before loading of the operation object on the pallet and/or before unloading of the operation object from the pallet;
deriving second data including information about the operation object and information about the pallet after loading of the operation object on the pallet and/or after unloading of the operation object from the pallet;
storing the first data and the second data;
creating or deriving, at least in part, a control sequence for loading of the operation object on the pallet and/or unloading of the operation object from the pallet based on the first data before loading of the operation object on the pallet and/or before unloading of the operation object from the pallet;
instructing the operation robot to, using an end effector of the robotic and based on the control sequence, execute a task of loading the operation object on the pallet and/or unloading the operation object from the pallet, and
before loading of the operation object on the pallet and/or unloading of the operation object from the pallet are/is actually finished, determining that the loading of the operation object on the pallet and/or the unloading of the operation object from the pallet are/is completed,
wherein creating or deriving the control sequence includes creating or deriving the control sequence based on the first data and/or the second data such that:
at least two operation objects having different stock keeping units from one another are loaded on the pallet, and
a plurality of operation objects having same or similar stock keeping units are placed on the pallet in a concentrated manner.

### Claim 8 (depends on 7)

The tangible, non-transitory computer-readable medium of claim 7, wherein:
the method further comprises calculating, based on the second data, an index indicating handling property for unloading of the operation object from the pallet, and
creating or deriving the control sequence includes creating or deriving, based on the index, the control sequence including identification of the operation object and the pallet loaded with the operation object.

### Claim 9 (depends on 7)

The tangible, non-transitory computer-readable medium of claim 7, wherein the method further comprises storing positional information about a plurality of the operation objects on the pallet as two-dimensional information or three-dimensional information for each operation object of the plurality or for each pallet.

### Claim 10 (depends on 7)

The tangible, non-transitory computer-readable medium of claim 7, wherein:
the operation object includes an identification code or tag attached to a side of the operation object,
creating or deriving the control sequence includes creating or deriving the control sequence for loading the operation object on the pallet so as to allow a sensor to image or read the identification code or tag after the operation object is loaded on the pallet, and
the sensor is configured for imaging of the operation object or of the pallet loaded with the operation object.

### Claim 11 (depends on 7)

The tangible, non-transitory computer-readable medium of claim 7, wherein the method further comprises deriving measured values or estimated values of the information about the operation object and/or of the information about the pallet.

### Claim 12

A method for controlling an operation robot adapted to manipulate operation objects to load the operation objects on a pallet and/or unload the operation objects from the pallet, wherein the method is performed by a controller having a data deriving unit, a data storage unit, and a robot control unit, and wherein the method comprises:
deriving first data using the data deriving unit, the first data including information about an operation object and information about the pallet, before loading of the operation object on the pallet and/or before unloading of the operation object from the pallet;
storing the first data using the data storage unit;
creating or deriving, at least in part and using the robot control unit and based on the first data, a control sequence for loading the operation object on the pallet and/or unloading the operation object from the pallet before loading of the operation object on the pallet and/or unloading of the operation object from the pallet;
instructing, using the robot control unit and based on the control sequence, the operation robot to execute, using an end effector of the robotic, a task of loading the operation object on the pallet and/or unloading the operation object from the pallet; and
determining that loading of the operation object on the pallet and/or unloading of the operation object from the pallet are/is completed, before the loading of the operation object on the pallet and/or the unloading of the operation object from the pallet are/is actually finished.

### Claim 13 (depends on 12)

The method of claim 12, further comprising:
deriving second data using the data deriving unit, the second data including information about the operation object and information about the pallet, after loading of the operation object on the pallet and/or after unloading of the operation object from the pallet; and
calculating, using the robot control unit and based on the second data, an index indicating handling property for unloading of the operation object from the pallet,
wherein deriving the control sequence includes deriving, based on the index, the control sequence including identification of the operation object and the pallet loaded with the operation object.

### Claim 14 (depends on 12)

The method of claim 12, further comprising storing, using the data storage unit, positional information about a plurality of the operation objects on the pallet as two-dimensional information or three-dimensional information for each operation object of the plurality of the operation objects or for each pallet.

### Claim 15 (depends on 12)

The method of claim 12, wherein:
the operation objection includes an identification code or tag attached to a side of the operation objection, and
creating or deriving the control sequence includes creating or deriving the control sequence for loading the operation object on the pallet so as to allow a sensor to image or read the identification code or tag after the operation object has been loaded on the pallet.

### Claim 16 (depends on 12)

The method of claim 12, further comprising deriving, using the data deriving unit, measured values or estimated values of the information about the operation object and/or of the information about the pallet.

### Claim 17 (depends on 12)

The method of claim 12, further comprising:
identifying a robotic system that includes the operation robot and a region related to the robotic system; and
calculating tasks based on the control sequence,
wherein the tasks include (i) a task related to transportation of the pallet and manipulation of the operation object by the robotic system and (ii) a plurality of tasks to be executed by the robotic system across adjacent and/or overlapping regions.