# US 11,897,706 B2 · Robotic system with zone-based control

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,897,706 B2
- Document ID: US11897706B2
- Status: Granted
- Kind: B2
- Application number: 17/217,467
- Filed: 2021-03-30
- Published: 2022-10-06
- Granted: 2024-02-13
- Earliest claimed: 2021-03-30
- Assignees: Dexterity, Inc.
- Applicants: Dexterity, Inc.
- Inventors: Zhouwen Sun; Cuthbert Sun; Andrew Nguyen; Samuel Tay; Emilie Peck; Kevin Jose Chavez; Robert Holmberg
- Web page: https://patentgrep.com/patent/US11897706B2

## CPC

- B25J 19/02: Sensing devices
- B65G 47/905: {Control arrangements}
- B25J 5/02: travelling along a guideway
- B25J 9/0084: {comprising a plurality of manipulators}
- B25J 9/1633: {compliant, force, torque control, e.g. combined with position control}
- B25J 9/1651: {acceleration, rate control}
- B25J 9/1661: {characterised by task planning, object-oriented languages}
- B25J 9/1666: {Avoiding collision or forbidden zones}
- B25J 9/1676: {Avoiding collision or forbidden zones}
- B25J 9/1679: {characterised by the tasks executed}
- B25J 9/1687: {Assembly, peg and hole, palletising, straight line, weaving pattern movement}
- B25J 9/1697: {Vision controlled systems}
- B25J 13/089: {Determining the position of the robot with reference to its environment}
- B25J 19/06: Safety devices
- B65G 1/1378: {the orders being assembled on fixed commissioning areas remote from the storage areas}
- B65G 61/00: Use of pick-up or transfer devices or of manipulators for stacking or de-stacking articles not otherwise provided for
- F16P 3/08: in connection with the locking of doors, covers, guards, or like members giving access to moving machine parts
- F16P 3/142: {using image capturing devices}
- F16P 3/144: {using light grids}
- F16P 3/147: {using electro-magnetic technology, e.g. tags or radar}
- B65G 2203/0283: Position of the load carrier
- B65G 2203/041: Camera
- B65G 2207/40: Safety features of loads, equipment or persons
- G05B 2219/39529: Force, torque sensor in wrist, end effector
- G05B 2219/40032: Peg and hole insertion, mating and joining, remote center compliance
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- B65G: TRANSPORT OR STORAGE DEVICES, e.g. CONVEYORS FOR LOADING OR TIPPING, SHOP CONVEYOR SYSTEMS OR PNEUMATIC TUBE CONVEYORS (packaging B65B; handling thin or filamentary materials, e.g. paper sheets or thread, B65H; cranes B66C; portable mobile or lifting or hauling appliances, e.g. hoists, B66D; devices for lifting or lowering goods for loading or unloading purposes, e.g. fork-lift trucks, B66F9/00; emptying bottles, jars, casks, barrels or similar containers, not otherwise provided for, B67C9/00; delivering or transferring liquids B67D; filling or discharging vessels for liquefied, solidified or compressed gases F17C; pipe-line systems for fluids F17D)
- F16P: SAFETY DEVICES IN GENERAL; {SAFETY DEVICES FOR PRESSES}
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A robotic line kitting system is disclosed. Data indicating for each of a plurality of operating zones comprising a workspace with which the robotic line kitting system is associated a corresponding safety state information is stored. The data is used to dynamically schedule and perform tasks associated with a higher level objective, including by operating the one or more robotic instrumentalities in one or more operating zones, if any, indicated by the data to currently be available to perform tasks using the one or more robotic instrumentalities and by not operating the one or more robotic instrumentalities in one or more operating zones, if any, indicated by the data to not currently be available to perform tasks using the one or more robotic instrumentalities.

## Claims

### Claim 1

A robotic line kitting system, comprising:
a memory configured to store data indicating, for each of a plurality of operating zones comprising a workspace with which the robotic line kitting system is associated, a corresponding state information indicating an extent to which that operating zone currently is or is not available to the robotic line kitting system to perform tasks using one or more robotic instrumentalities comprising the robotic line kitting system; and
a processor coupled to the memory and configured to:
use the data indicating, for each of the plurality of operating zones, the extent to which that operating zone currently is or is not available to the robotic line kitting system to perform tasks using the one or more robotic instrumentalities to dynamically schedule and perform tasks associated with a higher level objective, including by operating the one or more robotic instrumentalities in one or more operating zones, if any, indicated by said data to currently be available to perform tasks using the one or more robotic instrumentalities and by not operating the one or more robotic instrumentalities in one or more operating zones, if any, indicated by said data to not currently be available to perform tasks using the one or more robotic instrumentalities;
receive from a first sensor associated with a first operating zone of the plurality of operating zones, first operating zone state information associated with the first operating zone of the plurality of operating zones, wherein the first sensor is located at an entrance to the first operating zone;
receive from a second sensor associated with the first operating zone of the plurality of operating zones, second operating zone state information associated with the first operating zone, wherein the second sensor is located in the first operating zone at a distance offset from the first sensor;
in response to receiving the first operating zone state information associated with the first operating zone and the second operating zone state information associated with the first operating zone, update the corresponding state information associated with the first operating zone to indicate that the first operating zone is not available to the robotic line kitting system; and
in response to the corresponding state information associated with the first operating zone of the plurality of operating zones being updated to indicate that the first operating zone is not available to the robotic line kitting system, alter autonomous behavior of the one or more robotic instrumentalities to ensure safety of any human worker in the first operating zone, wherein the autonomous behavior of the one or more robotic instrumentalities is altered based on whether the first operating zone state information associated with the first operating zone indicates a presence of a human worker in the first operating zone and the second operating zone state information associated with the first operating zone does not indicate the presence of the human worker in the first operating zone or whether the first operating zone state information associated with the first operating zone and the second operating zone state information associated with the first operating zone indicates the presence of the human worker in the first operating zone.

### Claim 2 (depends on 1)

The robotic line kitting system of claim 1, wherein the processor is further configured to store in said memory said data indicating for each of the plurality of operating zones the extent to which that operating zone currently is or is not available to the robotic line kitting system to perform tasks using the one or more robotic instrumentalities.

### Claim 3 (depends on 2)

The robotic line kitting system of claim 2, wherein the processor is further configured to determine based on data received from one or more sensors said data indicating for each of the plurality of operating zones the extent to which that operating zone currently is or is not available to the robotic line kitting system to perform tasks using the one or more robotic instrumentalities.

### Claim 4 (depends on 3)

The robotic line kitting system of claim 3, wherein the first sensor and the second sensor are configured to detect access by the human worker to one or more of said operating zones.

### Claim 5 (depends on 3)

The robotic line kitting system of claim 3, wherein said one or more sensors includes data provided by a safety module comprising a first robotic instrumentality of the one or more robotic instrumentalities comprising the robotic line kitting system.

### Claim 6 (depends on 5)

The system of claim 5, wherein the first robotic instrumentality comprises a robotic arm and the safety module provides as output data indicating for each of one or more operable elements comprising the robotic arm one or both of a position in three-dimensional space of the operable element and data indicating whether all or part of the operable element has entered or will enter a location associated with one or more of said operating zones.

### Claim 7 (depends on 1)

The robotic line kitting system of claim 1, wherein the processor is further configured to cause a robotic instrumentality included in said one or more robotic instrumentalities to be operated in or adjacent to the first operating zone in a first mode associated with a first safety state of the first operating zone as indicated by a corresponding data included in said stored data.

### Claim 8 (depends on 7)

The robotic line kitting system of claim 7, wherein the first safety state is associated with presence of a human worker in the first operating zone and the first mode comprises a pause stop mode in which the robotic instrumentality is brought to a stop in a controlled manner.

### Claim 9 (depends on 7)

The robotic line kitting system of claim 7, wherein the first safety state is associated with presence of the human worker in the first operating zone and the first mode comprises a safe transit mode in which the robotic instrumentality is moved through or by the first operating zone at a reduced speed.

### Claim 10 (depends on 9)

The robotic line kitting system of claim 9, wherein the safe transit mode further includes emitting an audible warning sound.

### Claim 11 (depends on 9)

The robotic line kitting system of claim 9, wherein the safe transit mode further includes maintaining one or more operable elements of the robotic instrumentality in a stowed position.

### Claim 12 (depends on 1)

The robotic line kitting system of claim 1, wherein the processor is further configured to detect a change in said state information for a given operating zone by which the operating zone has changed from being currently available to the robotic line kitting system to perform tasks using the one or more robotic instrumentalities comprising the robotic line kitting system to a state in which the given operating zone is not currently available to the robotic line kitting system to perform tasks using the one or more robotic instrumentalities.

### Claim 13 (depends on 12)

The robotic line kitting system of claim 12, wherein the processor is further configured to determine a set of upcoming tasks scheduled to be performed at least in part in a region associated with the given operating zone and to revise a plan dynamically to cancel or modify the determined set of upcoming tasks to avoid operating in the given operating zone in a manner not permitted by a detected current state of the given operating zone.

### Claim 14 (depends on 13)

The robotic line kitting system of claim 13, wherein the processor is configured to revise the plan in a manner that enables the robotic line kitting system to continue working in one or more operating zones other than the given operating zone.

### Claim 15

A method to control a robotic line kitting system, comprising:
storing data indicating, for each of a plurality of operating zones comprising a workspace with which the robotic line kitting system is associated, a corresponding state information indicating an extent to which that operating zone currently is or is not available to the robotic line kitting system to perform tasks using one or more robotic instrumentalities comprising the robotic line kitting system;
using the data indicating, for each of the plurality of operating zones, the extent to which that operating zone currently is or is not available to the robotic line kitting system to perform tasks using the one or more robotic instrumentalities to dynamically schedule and perform tasks associated with a higher level objective, including by operating the one or more robotic instrumentalities in one or more operating zones, if any, indicated by said data to currently be available to perform tasks using the one or more robotic instrumentalities and by not operating the one or more robotic instrumentalities in one or more operating zones, if any, indicated by said data to not currently be available to perform tasks using the one or more robotic instrumentalities;
receiving from a first sensor associated with a first operating zone of the plurality of operating zones, first operating zone state information associated with the first operating zone of the plurality of operating zones, wherein the first sensor is located at an entrance to the first operating zone;
receiving from a second sensor associated with the first operating zone of the plurality of operating zones, second operating zone state information associated with the first operating zone, wherein the second sensor is located in the first operating zone at a distance offset from the first sensor;
in response to receiving the first operating zone state information associated with the first operating zone and the second operating zone state information associated with the first operating zone, updating the corresponding state information associated with the first operating zone to indicate that the first operating zone is not available to the robotic line kitting system; and
in response to the corresponding state information associated with the first operating zone of the plurality of operating zones being updated to indicate that the first operating zone is not available to the robotic line kitting system, altering autonomous behavior of the one or more robotic instrumentalities to ensure safety of any human worker in the first operating zone, wherein the autonomous behavior of the one or more robotic instrumentalities is altered based on whether the first operating zone state information associated with the first operating zone indicates a presence of a human worker in the first operating zone and the second operating zone state information associated with the first operating zone does not indicate the presence of the human worker in the first operating zone or whether the first operating zone state information associated with the first operating zone and the second operating zone state information associated with the first operating zone indicates the presence of the human worker in the first operating zone.

### Claim 16 (depends on 15)

The method of claim 15, further comprising determining based on data received from one or more sensors said data indicating for each of the plurality of operating zones the extent to which that operating zone currently is or is not available to the robotic line kitting system to perform tasks using the one or more robotic instrumentalities.

### Claim 17 (depends on 16)

The method of claim 16, wherein said one or more sensors include the first sensor configured to detect access by the human worker to one or more of said operating zones.

### Claim 18 (depends on 17)

The method of claim 17, wherein said one or more sensors includes data provided by a safety module comprising a first robotic instrumentality of the one or more robotic instrumentalities comprising the robotic line kitting system.

### Claim 19 (depends on 18)

The method of claim 18, wherein the first robotic instrumentality comprises a robotic arm and the safety module provides as output data indicating for each of one or more operable elements comprising the robotic arm one or both of a position in three-dimensional space of the operable element and data indicating whether all or part of the operable element has entered or will enter a location associated with one or more of said operating zones.

### Claim 20

A computer program product to control a robotic line kitting system, the computer program product being embodied in a non-transitory computer readable medium and comprising computer instructions for:
storing data indicating, for each of a plurality of operating zones comprising a workspace with which the robotic line kitting system is associated, a corresponding state information indicating an extent to which that operating zone currently is or is not available to the robotic line kitting system to perform tasks using one or more robotic instrumentalities comprising the robotic line kitting system;
using the data indicating, for each of the plurality of operating zones, the extent to which that operating zone currently is or is not available to the robotic line kitting system to perform tasks using the one or more robotic instrumentalities to dynamically schedule and perform tasks associated with a higher level objective, including by operating the one or more robotic instrumentalities in one or more operating zones, if any, indicated by said data to currently be available to perform tasks using the one or more robotic instrumentalities and by not operating the one or more robotic instrumentalities in one or more operating zones, if any, indicated by said data to not currently be available to perform tasks using the one or more robotic instrumentalities;
receiving from a first sensor associated with a first operating zone of the plurality of operating zones, first operating zone state information associated with the first operating zone of the plurality of operating zones, wherein the first sensor is located at an entrance to the first operating zone;
receiving from a second sensor associated with the first operating zone of the plurality of operating zones, second operating zone state information associated with the first operating zone, wherein the second sensor is located in the first operating zone at a distance offset from the first sensor;
in response to receiving the first operating zone state information associated with the first operating zone and the second operating zone state information associated with the first operating zone, updating the corresponding state information associated with the first operating zone to indicate that the first operating zone is not available to the robotic line kitting system; and
in response to the corresponding state information associated with the first operating zone of the plurality of operating zones being updated to indicate that the first operating zone is not available to the robotic line kitting system, altering autonomous behavior of the one or more robotic instrumentalities to ensure safety of any human worker in the first operating zone, wherein the autonomous behavior of the one or more robotic instrumentalities is altered based on whether the first operating zone state information associated with the first operating zone indicates a presence of a human worker in the first operating zone and the second operating zone state information associated with the first operating zone does not indicate the presence of the human worker in the first operating zone or whether the first operating zone state information associated with the first operating zone and the second operating zone state information associated with the first operating zone indicates the presence of the human worker in the first operating zone.