# US 10,870,204 B2 · Robotic system control method and controller

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,870,204 B2
- Document ID: US10870204B2
- Status: Granted
- Kind: B2
- Application number: 16/752,582
- Filed: 2020-01-24
- Published: 2020-07-30
- Granted: 2020-12-22
- Earliest claimed: 2019-01-25
- Assignees: MUJIN, Inc.
- Applicants: MUJIN, Inc.
- Inventors: Rosen Nikolaev Diankov; Yoshiki Kanemoto; Denys Kanunikov
- Web page: https://patentgrep.com/patent/US10870204B2

## CPC

- B25J 9/1664: {characterised by motion, path, trajectory planning}
- B25J 9/1661: {characterised by task planning, object-oriented languages}
- B25J 9/1697: {Vision controlled systems}
- G01S 17/89: for mapping or imaging
- G05B 19/418: Total factory control, i.e. centrally controlling a plurality of machines, e.g. direct or distributed numerical control [DNC], flexible manufacturing systems [FMS], integrated manufacturing systems [IMS] or computer integrated manufacturing [CIM]
- G06T 7/73: using feature-based methods
- G05B 2219/39106: Conveyor, pick up article, object from conveyor, bring to test unit, place it
- G05B 2219/39107: Pick up article, object, measure, test it during motion path, place it
- G05B 2219/40053: Pick 3-D object from pile of objects
- Y02P 90/02: Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- Y02P: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS

## Abstract

The present disclosure provides a control method of a robotic system that implements a high degree of cooperation between units including a robot and increases a storage efficiency of an operation object sufficiently. A control method includes: deriving an approach location at which the end effector grips an operation object; deriving a scan location for scanning an identifier of the operation object; and based on the approach location and the scan location, creating or deriving a control sequence to instruct the robot to execute the control sequence. The control sequence includes (1) gripping the operation object from a start location; (2) scanning an identifier of the operation object with a scanner located between the start location and a task location; (3) temporarily releasing the operation object from the end effector and regripping the operation object by the end effector to be shifted, at a shift location, when a predetermined condition is satisfied; and (4) moving the operation object to the task location.

## Claims

### Claim 1

A control method of a robotic system that includes a robot having a robotic arm and an end effector, the method comprising:
deriving an approach location at which the end effector grips an operation object;
deriving a scan location for scanning an identifier of the operation object; and
based on the approach location and the scan location, creating or deriving a control sequence to instruct the robot to execute the control sequence,
wherein the control sequence includes:
gripping the operation object at a start location;
scanning identification information of the operation object with a scanner located between the start location and a task location;
temporarily releasing the operation object from the end effector and regripping the operation object by the end effector to be shifted, at a shift location, when a predetermined condition is satisfied;
moving the operation object to the task location,
setting the predetermined condition based on a condition that a storage efficiency of the operation object at the task location is increased when the operation object is shifted and a direction of gripping the operation object by the end effector is changed; and
calculating a first storage efficiency at the task location before shifting the operation object and a second storage efficiency at the task location after shifting the operation object.

### Claim 2 (depends on 1)

The method of claim 1, wherein the control sequence further includes:
deriving a height of the operation object; and
calculating the storage efficiency based on the height of the operation object.

### Claim 3 (depends on 2)

The method of claim 2, wherein the height of the operation object is calculated from a height location of a top surface of the operation object and a height location of a bottom surface of the operation object measured in a state of being gripped by the end effector.

### Claim 4 (depends on 2)

The method of claim 2, wherein the height of the operation object is measured when the operation object is scanned with the scanner.

### Claim 5 (depends on 1)

The method of claim 1, wherein the control sequence further includes,
temporarily releasing the operation object from the end effector by placing the operation object on a temporary placing table, at the shift location, when the predetermined condition is satisfied.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
obtaining imaging data representative of a pick-up area including the operation object;
determining an initial pose of the operation object based on the imaging data;
calculating a confidence measure in relation to an accuracy of the initial pose of the operation object; and
deriving the approach location and the scan location based on the confidence measure.

### Claim 7 (depends on 6)

The method claim 6,
wherein the control sequence further includes:
selectively calculating the approach location and the scan location according to a performance metric and/or a scan metric, based on a result of comparing the confidence measure to a sufficiency threshold, and
wherein the scan metric is related to a likelihood that the identifier of the operation object is not covered by the end effector, regardless of the confidence measure in relation to the initial pose of the operation object.

### Claim 8 (depends on 6)

The method of claim 6, wherein in a case where the confidence measure does not satisfy a sufficiency threshold, the approach location and the scan location are derived based on the scan metric or are derived based on the scan metric with prioritizing the scan metric over the performance metric.

### Claim 9 (depends on 6)

The method of claim 6, wherein in a case where the confidence measure satisfies a sufficiency threshold, the approach location and the scan location are derived based on the performance metric.

### Claim 10 (depends on 1)

The method of claim 1, wherein the control sequence further includes:
deriving a first scan location for providing identification information of the operation object to the scanner;
deriving a second scan location for providing alternative identification information of the operation object to the scanner; and
moving the operation object to the task location and ignoring the second scan location in a case where a scan result indicates a successful scan after the operation object is moved to the first scan location.

### Claim 11

A non-transitory computer-readable medium storing processor instructions for performing a control method of a robotic system that includes a robot having a robotic arm and an end effector,
the processor instructions including:
an instruction for deriving an approach location at which the end effector grips an operation object;
an instruction for deriving a scan location for scanning identification information of the operation object; and
an instruction for creating or deriving a control sequence to instruct the robot to execute the control sequence, based on the approach location and the scan location,
wherein the control sequence includes the following:
gripping the operation object at a start location;
scanning an identifier of the operation object with a scanner located between the start location and a task location;
temporarily releasing the operation object from the end effector and regripping the operation object by the end effector to be shifted, at a shift location, when a predetermined condition is satisfied;
moving the operation object to the task location,
setting the predetermined condition based on a condition that a storage efficiency of the operation object at the task location is increased when the operation object is shifted and a direction of gripping the operation object by the end effector is changed; and
calculating a first storage efficiency at the task location before shifting the operation object and a second storage efficiency at the task location after shifting the operation object.

### Claim 12 (depends on 11)

The non-transitory computer-readable medium of claim 11, wherein the control sequence further includes:
deriving a height of the operation object; and
calculating the storage efficiency based on the height of the operation object.

### Claim 13 (depends on 12)

The non-transitory computer-readable medium of claim 12, wherein the height of the operation object is calculated from a height location of a top surface of the operation object and a height location of a bottom surface of the operation object measured in a state of being gripped by the end effector.

### Claim 14

A controller of a robotic system that includes a robot having a robotic arm and an end effector, the controller comprising:
a communication device configured to communicate with a remote device; and
a processor coupled to the communication device, wherein the processor is configured to execute a control method, the control method comprising:
deriving an approach location at which the end effector grips an operation object;
deriving a scan location for scanning an identifier of the operation object; and
based on the approach location and the scan location, deriving a control sequence to instruct the robot to execute the control sequence,
wherein the control sequence includes:
gripping the operation object at a start location;
scanning identification information of the operation object with a scanner located between the start location and a task location;
calculating a first storage efficiency at the task location before shifting the operation object and a second storage efficiency at the task location after shifting the operation object;
temporarily releasing the operation object from the end effector and regripping the operation object by the end effector to be shifted, at a shift location, when a predetermined condition is satisfied, wherein the predetermined condition is set based on a condition that a storage efficiency of the operation object at the task location is increased when the operation object is shifted and a direction of gripping the operation object by the end effector is changed; and
moving the operation object to the task location.

### Claim 15 (depends on 14)

The controller of claim 14, wherein the control sequence further includes:
deriving a height of the operation object; and
calculating the storage efficiency based on the height of the operation object.

### Claim 16 (depends on 14)

The controller of claim 14, wherein the control sequence further includes generating commands and/or settings for temporarily releasing the operation object from the end effector by placing the operation object on a temporary placing table, at the shift location, when the predetermined condition is satisfied.

### Claim 17 (depends on 14)

The controller of claim 14, wherein the control method further includes:
obtaining imaging data representative of a pick-up area including the operation object;
determining an initial pose of the operation object based on the imaging data;
calculating a confidence measure in relation to an accuracy of the initial pose of the operation object; and
deriving the approach location and the scan location based on the confidence measure.

### Claim 18 (depends on 14)

The controller of claim 14, wherein the control sequence further includes:
deriving a first scan location for providing identification information of the operation object to the scanner;
deriving a second scan location for providing alternative identification information of the operation object to the scanner; and
generating commands and/or settings to move the operation object to the task location while bypassing the second scan location when a scan result indicates a successful scan after the operation object is moved to the first scan location.

### Claim 19 (depends on 14)

The controller of claim 14, wherein the control sequence further includes:
deriving a first scan location for providing identification information of the operation object to the scanner;
deriving a second scan location for providing alternative identification information of the operation object to the scanner; and
generating commands and/or settings to move the operation object to the second scan location when a scan result indicates a failed scan after the operation object is moved to the first scan location.

### Claim 20 (depends on 1)

The method of claim 1, wherein the control sequence further includes:
deriving a first scan location for providing identification information of the operation object to the scanner;
deriving a second scan location for providing alternative identification information of the operation object to the scanner; and
moving the operation object to the second scan location when a scan result indicates a failed scan after the operation object is moved to the first scan location.