# US 10,456,915 B1 · Robotic system with enhanced scanning mechanism

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,456,915 B1
- Document ID: US10456915B1
- Status: Granted
- Kind: B1
- Application number: 16/258,120
- Filed: 2019-01-25
- Granted: 2019-10-29
- Earliest claimed: 2019-01-25
- Assignees: MUJIN, Inc.
- Applicants: MUJIN, Inc.
- Inventors: Rosen Nikolaev Diankov
- Web page: https://patentgrep.com/patent/US10456915B1

## CPC

- B25J 9/1612: {characterised by the hand, wrist, grip control}
- B25J 9/1661: {characterised by task planning, object-oriented languages}
- B25J 9/1664: {characterised by motion, path, trajectory planning}
- B25J 9/1697: {Vision controlled systems}
- G06T 7/73: using feature-based methods
- G01S 17/89: for mapping or imaging
- G05B 2219/39543: Recognize object and plan hand shapes in grasping movements
- G05B 2219/40053: Pick 3-D object from pile of objects
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A method for operating a robotic system including determining an initial pose of a target object based on imaging data, wherein the initial pose is for estimating a resting orientation of the target object in a pickup area; calculating a confidence measure associated with the initial pose, wherein the confidence measure is for representing a likelihood of the initial pose being correct; and calculating a motion plan according to the confidence measure, the motion plan for executing a task based on picking up the target object from a start location, transferring the target object to a task location, and scanning one or more object identifiers between the start location and the task location.

## Claims

### Claim 1

A method for operating a robotic system, the method comprising:
receiving imaging data representing a pickup area including a target object;
determining an initial pose of the target object based on the imaging data;
calculating a confidence measure associated with the initial pose, wherein the confidence measure is indicative of a likelihood of the initial pose being accurate;
deriving, based on the confidence measure, an approach location and at least one scanning location, the approach location defining a location for the end-effector to contact and grip the target object, the scanning location defining a location for the end-effector to present one or more surfaces of the target object before one or more corresponding object scanners so as to scan the one or more object identifiers;
deriving, based on the derived approached location and the scanning location, a motion plan,
wherein the motion plan includes commands, settings, or a combination thereof, for operating a robotic arm and an end-effector to execute a task based on (1) picking up the target object from a start location, (2) transferring the target object to a task location, and (3) presenting one or more object identifiers to one or more object scanners located between the start location and the task location,
wherein deriving the motion plan includes:
comparing the confidence measure to a sufficiency threshold,
selectively calculating the approach location and the scanning location according to a performance metric and/or a scanning metric, based on an outcome of the comparison, wherein the scanning metric corresponds to a likelihood of at least one of the object identifiers remaining uncovered by the end-effector regardless of whether or not the initial pose is accurate; and
implementing the motion plan for operating one or more actuation devices to execute the task;
wherein:
deriving the approach location and the scanning location includes prioritizing the scanning metric before the performance metric when the confidence measure fails to satisfy the sufficiency threshold.

### Claim 2 (depends on 1)

The method of claim 1, wherein deriving the approach location and the scanning location includes deriving the approach location and the scanning location according to the scanning metric when the confidence measure fails to satisfy the sufficiency threshold.

### Claim 3 (depends on 1)

The method of claim 1, wherein deriving the approach location and the scanning location includes deriving the approach location and the scanning location based on the performance metric when the confidence measure satisfies the sufficiency threshold.

### Claim 4 (depends on 1)

The method of claim 1, wherein deriving the approach location and the scanning location includes:
calculating a set of available approach locations about the start location for the target object, wherein the set of available approach locations for placing the end-effector to contact and grip the target object without disturbing other objects;
calculating an exposed identifier set for each available approach location in the set of available approach locations, wherein the exposed identifier set represents corresponding locations of the object identifiers remaining uncovered with the end-effector at the corresponding approach location according to a first hypothesis of the initial pose being accurate;
calculating an alternative identifier set for each available approach location in the set of available approach locations, wherein the alternative identifier set represents locations of the object identifiers remaining uncovered with the end-effector at the corresponding approach location according to a second hypothesis of the initial pose being inaccurate; and
selecting the approach location, wherein the approach location is the available approach location having uncovered object identifiers in both the exposed identifier set and the alternative identifier set.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
identifying scanning fields corresponding to the object scanners;
calculating a set of candidate poses for the target object in or through the scanning fields, wherein each candidate pose presents multiple surfaces of the target object to the object scanners;
mapping the exposed identifier set and the alternative identifier set to each of the candidate poses;
comparing the exposed identifier set and the alternative identifier set for each candidate pose to the scanning fields;
identifying a scanning pose as a candidate pose presenting the object identifiers in the exposed identifier set and the alternative identifier set to the scanning fields; and
wherein deriving the motion plan includes:
deriving the motion plan establishing the scanning pose in or through the scanning fields.

### Claim 6 (depends on 5)

The method of claim 5, wherein:
identifying scanning fields includes identifying scanning fields oriented in opposite directions, orthogonal directions, or a combination thereof for scanning different surfaces of the target object in one pose; and
implementing the motion plan includes implementing the motion plan for transferring the target object in the scanning pose across the scanning fields in a direction orthogonal to orientations of the scanning fields.

### Claim 7 (depends on 4)

The method of claim 4, wherein deriving the motion plan includes calculating at least a first scanning location and a second scanning location, wherein the first scanning location is for presenting a location in the exposed identifier set to one of the object scanners and the second scanning location is for presenting a location in the alternative identifier set to one of the object scanners.

### Claim 8 (depends on 7)

The method of claim 7, wherein implementing the motion plan includes:
verifying a scanning result after implementing the motion plan up to the first scanning location; and
transferring the target object to the task location and ignoring the second scanning location when the scanning result represents a successful scan.

### Claim 9 (depends on 7)

The method of claim 7, wherein implementing the motion plan includes:
verifying a scanning result after implementing the motion plan up to the first scanning location; and
transferring the target object to the second scanning location when the scanning result represents a failed scan.

### Claim 10 (depends on 4)

The method of claim 4, further comprising:
comparing the exposed identifier set to the alternative identifier set to determine whether the exposed identifier set and the alternative identifier set include locations on opposing surfaces of the target object; and
selecting the approach location associated with a third surface orthogonal to the opposing surfaces.

### Claim 11 (depends on 10)

The method of claim 10, wherein deriving the approach location and the scanning location includes deriving the approach location and the scanning location without considering the scanning metric when the confidence measure satisfies the sufficiency threshold.

### Claim 12 (depends on 1)

The method of claim 1, wherein calculating the motion plan includes:
calculating a set of available approach locations about the start location for the target object, wherein the set of available approach locations represents positions for placing the end-effector to grip the target object without disturbing other objects;
calculating a candidate plan for each available approach location in the set of available approach locations, wherein the candidate plan includes a candidate scanning location;
calculating the performance metric for each candidate plan, wherein the performance metric corresponds to a throughput rate associated with a distance traversed by the target object, an estimated transfer duration, a number of commands or setting changes, a completion rate associated with the corresponding available approach location, or a combination thereof for placing the target object at the task location; and
selecting the approach location, wherein the approach location is an available approach location in the set corresponding to the highest performance metric.

### Claim 13 (depends on 1)

The method of claim 1, wherein determining the initial pose of the target object includes:
identifying a set of edges of an exposed surface of the target object in the imaging data;
estimating physical dimensions of the target object based on measuring the edges in the set and mapping the measurements into standard lengths; and
determining the initial pose based on identifying the exposed surface, including matching the estimated physical dimensions of the exposed surface to a set of known dimensions of object surfaces in master data.

### Claim 14 (depends on 13)

The method of claim 13, wherein:
determining the initial pose includes identifying a pair of opposing surfaces matching the estimated physical dimensions; and
when the confidence measure fails to satisfy the sufficiency threshold, calculating the motion plan that accounts for an opposing pose in addition to the initial pose, wherein the opposing pose and the initial pose correspond to the pair of opposing surfaces.

### Claim 15 (depends on 13)

The method of claim 13, wherein determining the initial pose includes determining the initial pose based on identifying an exposed surface, an orientation thereof, or a combination thereof using a visible mark on the exposed surface, including matching the visible mark to a set of known marks on object surfaces in master data.

### Claim 16 (depends on 1)

The method of claim 1, wherein:
implementing the motion plan includes determining a scanning result; and
further comprising:
halting the motion plan when the scanning result indicates a failed scan; and
implementing a failure resolution for notifying an operator, for placing the target object at a location different from the start location and the task location, or a combination thereof.

### Claim 17

A method for operating a robotic system, the method comprising:
identifying one or more scanning fields, wherein each of the scanning fields represent a space adjacent to an object scanner;
determining an initial pose of a target object based on imaging data, wherein the target object includes one or more object identifiers;
calculating a first estimated location of one of the object identifiers according to a first hypothesis of the initial pose being accurate;
calculating a second estimated location of one of the object identifiers according to a second hypothesis of the initial pose being inaccurate;
and
deriving a motion plan that places the first estimated location and the second estimated location in one or more of the scanning fields, wherein the motion plan includes actuation device commands, actuation device settings, or a combination thereof for operating a robotic arm and an end-effector to (1) pick up the target object, (2) transfer the target object, and (3) present the one or more object identifiers to one or more object scanners during the transfer, wherein:
the motion plan is derived based on an approached location and a scanning location, the approach location defining a location for the end-effector to contact and grip the target object, the scanning location defining a location for the end-effector to present one or more surfaces of the target object before one or more corresponding object scanners so as to scan the one or more object identifiers,
the approached location and the scanning location are selectively calculated according to a performance metric and/or a scanning metric, wherein the scanning metric corresponds to a likelihood of at least one of the object identifiers remaining uncovered by the end-effector regardless of whether or not the initial pose is accurate;
wherein:
deriving the approach location and the scanning location includes prioritizing the scanning metric before the performance metric when the confidence measure fails to satisfy the sufficiency threshold.

### Claim 18

A tangible, non-transitory computer-readable medium having processor instructions stored thereon that, when executed by a robotic system via one or more processors thereof, cause the robotic system to:
receive imaging data representing a pickup area including a target object;
determine an initial pose of the target object based on the imaging data;
calculate a confidence measure associated with the initial pose, wherein the confidence measure is indicative of a likelihood of the initial pose being accurate;
derive, based on the confidence measure, an approach location and at least one scanning location, the approach location defining a location for the end-effector to contact and grip the target object, the scanning location defining a location for the end-effector to present one or more surfaces of the target object before one or more corresponding object scanners so as to scan the one or more object identifiers;
derive, based on the derived approached location and the scanning location, a motion plan,
wherein the motion plan includes commands, settings, or a combination thereof, for operating a robotic arm and an end-effector to execute a task based on (1) picking up the target object from a start location, (2) transferring the target object to a task location, and (3) presenting one or more object identifiers to one or more object scanners located between the start location and the task location,
wherein the motion plan is derived based on:
comparing the confidence measure to a sufficiency threshold,
selectively calculating the approach location and the scanning location according to a performance metric and/or a scanning metric, based on an outcome of the comparison, wherein the scanning metric corresponding to a likelihood of at least one of the object identifiers remaining uncovered by the end-effector regardless of whether or not the initial pose is accurate; and
implement the motion plan for operating one or more actuation devices to execute the task;
wherein:
the approach location and the scanning location are derived based on prioritizing the scanning metric before the performance metric when the confidence measure fails to satisfy the sufficiency threshold.

### Claim 19 (depends on 18)

The tangible, non-transitory computer-readable medium of claim 18, wherein the approach location and the scanning location is derived based on:
calculating a set of available approach locations about the start location for the target object, wherein the set of available approach locations for placing the end-effector to contact and grip the target object without disturbing other objects;
calculating an exposed identifier set for each available approach location in the set of available approach locations, wherein the exposed identifier set represents corresponding locations of the object identifiers remaining uncovered with the end-effector at the corresponding approach location according to a first hypothesis of the initial pose being accurate;
calculating an alternative identifier set for each available approach location in the set of available approach locations, wherein the alternative identifier set represents locations of the object identifiers remaining uncovered with the end-effector at the corresponding approach location according to a second hypothesis of the initial pose being inaccurate; and
selecting as the approach location the available approach location having uncovered object identifiers in both the exposed identifier set and the alternative identifier set.