# US 12,005,578 B2 · Real-time real-world reinforcement learning systems and methods

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,005,578 B2
- Document ID: US12005578B2
- Status: Granted
- Kind: B2
- Application number: 16/560,761
- Filed: 2019-09-04
- Published: 2020-03-05
- Granted: 2024-06-11
- Earliest claimed: 2018-09-04
- Assignees: Ocado Innovations Limited
- Applicants: Ocado Innovation Limited
- Inventors: Ashique Rupam Mahmood; Brent J. Komer; Dmytro Korenkevych
- Web page: https://patentgrep.com/patent/US12005578B2

## CPC

- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- B25J 9/161: {Hardware, e.g. neural networks, fuzzy logic, interfaces, processor}
- G06F 18/2178: {based on feedback of a supervisor}
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 20/00: Machine learning
- G06V 10/7784: {based on feedback from supervisors}
- G06V 10/96: Management of image or video recognition tasks
- G05B 2219/40499: Reinforcement learning algorithm
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A reinforcement learning architecture for facilitating reinforcement learning in connection with operation of an external real-time system that includes a plurality of devices operating in a real-world environment. The reinforcement learning architecture includes a plurality of communicators, a task manager, and a reinforcement learning agent that interact with each other to effectuate a policy for achieving a defined objective in the real-world environment. Each of the communicators receives sensory data from a corresponding device and the task manager generates a joint state vector based on the sensory data. The reinforcement learning agent generates, based on the joint state vector, a joint action vector, which the task manager parses into a plurality of actuation commands. The communicators transmit the actuation commands to the plurality of devices in the real-world environment.

## Claims

### Claim 1

A method, comprising:
establishing, via one or more communication interfaces, communications with an external real-time system comprising a plurality of robots;
obtaining, by a set of first processes executing on a computer system, a plurality of state data respectively indicating states of the plurality of robots;
generating, by a second process executing on the computer system, a joint state vector based on the plurality of state data stored in a plurality of first buffers;
transitioning, by a third process executing on the computer system, from a suspended state, in which the third process operates for a predefined period of time and does not obtain or process another joint state vector, to an active state;
generating, by the third process during the active state, a joint action vector based on the joint state vector according to a defined policy of the third process;
transitioning, by the third process in response to generating the joint action vector, back to the suspended state;
parsing, by the second process, the joint action vector into a plurality of actuation commands respectively indicating operations for each of the plurality of robots to perform; and
controlling, by each of the set of first processes, at least one robot of the plurality of robots to take an action, wherein controlling the at least one robot comprises:
transmitting, via the one or more communication interfaces, respective actuation commands of the plurality of actuation commands to a corresponding robot of the plurality of robots.

### Claim 2 (depends on 1)

The method of claim 1, comprising:
loading, by the set of first processes, individual state data of the plurality of state data into a corresponding buffer of the plurality of first buffers;
obtaining, by the second process, the plurality of state data from the plurality of first buffers; and
loading, by the second process, the joint state vector into a second buffer.

### Claim 3 (depends on 2)

The method of claim 2, wherein each of the plurality of first buffers is a circular buffer.

### Claim 4 (depends on 1)

The method of claim 1, comprising:
loading, by the third process during the active state, the joint action vector into a third buffer;
obtaining, by the second process, the joint action vector from the third buffer;
loading, by the second process, the plurality of actuation commands into corresponding buffers of a plurality of fourth buffers; and
obtaining, by the set of first processes, the actuation commands from the plurality of fourth buffers.

### Claim 5 (depends on 4)

The method of claim 4, wherein each of the plurality of fourth buffers is a circular buffer.

### Claim 6 (depends on 1)

The method of claim 1, wherein the joint state vector is generated based on a defined objective of the second process, the defined objective to be achieved in a physical environment of the external real-time system.

### Claim 7 (depends on 1)

The method of claim 1, comprising:
generating, by the second process, information having a value representative of a state of a physical environment of the external real-time system relative to a defined objective, wherein the information is included in the joint state vector.

### Claim 8 (depends on 1)

The method of claim 1, wherein the defined policy maps a state of the external real-time system to a corresponding action to be performed by a robot of the plurality of robots.

### Claim 9 (depends on 1)

The method of claim 1, wherein the third process operates in the suspended state for a defined period of time prior to transitioning to the active state, and the third process transitions from the active state to the suspended state as a result of generating the joint action vector.

### Claim 10 (depends on 1)

The method of claim 1, comprising:
updating, by the third process, the defined policy based on the joint state vector.

### Claim 11 (depends on 10)

The method of claim 10, wherein the defined policy is updated during a defined period of time in which the third process is in the suspended state.

### Claim 12 (depends on 1)

The method of claim 1, wherein the set of first processes, the second process, and the third process are executing on a distributed computing system remotely located from the external real-time system.

### Claim 13 (depends on 1)

The method of claim 1, wherein the set of first processes is executing on one or more first processors of the computer system, the second process is executing on a second processor of the computer system, and the third process is executing on a third processor of the computer system.

### Claim 14

A system, comprising:
one or more communication interfaces configured to communicate over one or more networks;
one or more processors coupled to the one or more communication interfaces;
memory storing a set of instructions that cause the one or more processors to establish, via the one or more communication interfaces, communications with an external real-time system comprising a plurality of robots and that cause the one or more processors to execute processes that include:
a first communicator configured to receive first state data that indicates a state of a first robot of the plurality of robots, and cause the first robot to be controlled to take an action by transmitting a first actuation command to the first robot via the one or more communication interfaces;
a second communicator configured to receive second state data that indicates a state of a second robot of the plurality of robots, and to cause the second robot to be controlled to take an action by transmitting a second actuation command to the second robot via the one or more communication interfaces;
a task manager configured to:
generate a joint state vector based on the first state data and the second state data; and
parse a joint action vector into the first actuation command and the second actuation command; and
a reinforcement learning agent configured to:
transition from a suspended state, in which the reinforcement learning agent operates for a predefined period of time and does not obtain or process another joint state vector, to an active state;
generate, in the active state, the joint action vector based on the joint state vector; and
transition, in response to generating the joint action vector, back to the suspended state.

### Claim 15 (depends on 14)

The system of claim 14, wherein, as a result of execution of the set of instructions by the one or more processors,
the first communicator is configured to store the first state data in a first buffer;
the second communicator is configured to store the second state data in a second buffer;
the task manager is configured to obtain the first state data from the first buffer and the second state data from the second buffer; and
the task manager is configured to store the joint state vector into a third buffer.

### Claim 16 (depends on 14)

The system of claim 14, wherein, as a result of execution of the set of instructions by the one or more processors,
the reinforcement learning agent is configured to, in the active state, obtain the joint state vector from a first buffer and store the joint action vector into a second buffer;
the task manager is configured to obtain the joint action vector from the second buffer and store the first actuation command into a third buffer and store the second actuation command into a fourth buffer;
the first communicator is configured to read the first actuation command from the third buffer; and
the second communicator is configured to read the second actuation command from the fourth buffer.

### Claim 17 (depends on 14)

The system of claim 14, wherein, as a result of execution of the set of instructions by the one or more processors, the task manager is configured to generate the joint state vector based on a defined objective that is to be achieved in a physical environment of the external real-time system.

### Claim 18 (depends on 14)

The system of claim 14, wherein, as a result of execution of the set of instructions by the one or more processors, the task manager is configured to generate a value, to be included in the joint state vector, representative of a change in state of a physical environment of the external real-time system relative to a defined objective, the change in state as a result of a previous joint action vector generated by the reinforcement learning agent.

### Claim 19 (depends on 14)

The system of claim 14, wherein, as a result of execution of the set of instructions by the one or more processors, the reinforcement learning agent is configured to operate in the suspended state for a defined period of time prior to transitioning to the active state, and configured to transition from the active state to the suspended state as a result of generating the joint action vector.

### Claim 20 (depends on 14)

The system of claim 14, wherein the one or more processors are of a distributed computer system remotely located from the external real-time system.

### Claim 21 (depends on 14)

The system of claim 14, wherein the task manager is a first process that executes on a first processor of the one or more processors, and the reinforcement learning agent is a second process the executes on a second processor of the one or more processors.

### Claim 22

At least one non-transitory computer-readable media storing a set of instructions that, as a result of execution by one or more processors of a computer system, cause the one or more processors to implement a plurality of processes that include:
a first communicator process that receives first state data that indicates a state of a first robot operating in a physical environment, and causes the first robot to be controlled to take an action by transmitting a first actuation command to the first robot;
a second communicator process that receives second state data that indicates a state of a second robot operating in the physical environment, and causes the second robot to be controlled to take an action by transmitting a second actuation command to the second robot;
a task manager process that:
generates a joint state vector based on the first state data and the second state data; and
parses a joint action vector into the first actuation command and the second actuation command; and
a reinforcement learning agent process that:
transitions from a suspended state, in which the reinforcement learning agent operates for a predefined period of time and does not obtain or process another joint state vector, to an active state;
generates, in the active state, the joint action vector based on the joint state vector; and
transition, in response to generating the joint action vector, back to the suspended state.

### Claim 23 (depends on 22)

The at least one non-transitory computer-readable media of claim 22, wherein the reinforcement learning agent process implemented by the one or more processors as a result of executing the set of instructions further:
operates in the suspended state for a defined period of time prior to transitioning to the active state, and transitions from the active state to the suspended state as a result of generating the joint action vector.

### Claim 24 (depends on 22)

The at least one non-transitory computer-readable media of claim 22, wherein the set of instructions, as a result of execution by the one or more processors, cause the one or more processors to:
execute the first communicator process on a set of first processors;
execute the second communicator process on the set of first processors;
execute the task manager process on a second processor; and
execute the reinforcement learning agent process on a third processor.

### Claim 25 (depends on 22)

The at least one non-transitory computer-readable media of claim 22, wherein the at least one non-transitory computer-readable media is of a distributed computing system remotely located from the physical environment.