# US 2021/0264795 A1 · Fault-tolerant control system and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0264795 A1
- Document ID: US20210264795A1
- Status: Published application
- Kind: A1
- Application number: 16/796,057
- Filed: 2020-02-20
- Published: 2021-08-26
- Earliest claimed: 2020-02-20
- Assignees: Prowler ,Io Limited
- Applicants: Prowler .Io Limited
- Inventors: David MGUNI
- Web page: https://patentgrep.com/patent/US20210264795A1

## CPC

- G06N 3/092: Reinforcement learning
- G06Q 10/063: Operations research, analysis or management
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/08: Learning methods
- G06N 3/094: Adversarial learning
- G06Q 50/40: Business processes related to the transportation industry (shipping G06Q10/083)
- G07C 5/008: {communicating information to a remotely located station (transmission systems for measured values G08C)}
- G07C 5/0816: {Indicating performance data, e.g. occurrence of a malfunction}
- G08G 5/26: Transmission of traffic-related information between aircraft and ground stations
- B64U 2201/00: UAVs characterised by their flight controls
- B64U 2201/10: autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
- G08G 5/22: located on the ground
- G08G 5/53: for cruising
- G08G 5/55: for a single aircraft
- G08G 5/57: for unmanned aircraft
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G07C: TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})
- B64U: UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR

## Abstract

A computer implemented method includes receiving data indicative of one or more experience tuples each comprising a first observation including a first location of an unmanned aerial vehicle, UAV, a first flight action performed by the UAV in dependence on the first observation, a reward associated with the performance of the first flight action, and a second observation including a second location of the UAV following the performance of the first action. For each of the one or more experience tuples, the method includes, at a computing system: processing the first observation, using a value estimator with current parameter values, to determine a first estimated return for the first flight action following the first observation; processing the second observation, using a target value estimator with an identical architecture to the value estimator, to determine a set of candidate estimated returns, each of the set of candidate estimated returns corresponding to a respective one of a set of candidate second flight actions following the second observation; determining a greatest of the determined candidate estimated returns; determining a terminal reward associated with a triggering of a failure condition corresponding to a failure of a physical component of the UAV following the UAV visiting the second location of the UAV; determining, using the determined terminal reward and the greatest candidate estimated return, a second estimated return for the first flight action following the first observation, accounting for an intervention by an adversarial stopping agent arranged to trigger the failure condition when predetermined stopping criteria are satisfied; and updating the current parameter values of the value estimator in dependence upon a difference between the first estimated return and the second estimated return. After being sequentially updated in accordance with each of the one or more experience tuples, the current parameter values of the value estimator are trained parameter values.

## Claims

### Claim 1

A computer-implemented method comprising:
receiving data indicative of one or more experience tuples each comprising a first observation including a first location of an unmanned aerial vehicle, UAV, a first flight action performed by the UAV in dependence on the first observation, a reward associated with the performance of the first flight action, and a second observation including a second location of the UAV following the performance of the first action;
for each of the one or more experience tuples, at a computing system:
processing the first observation, using a value estimator with current parameter values, to determine a first estimated return for the first flight action following the first observation;
processing the second observation, using a target value estimator with an identical architecture to the value estimator, to determine a set of candidate estimated returns, each of the set of candidate estimated returns corresponding to a respective one of a set of candidate second flight actions following the second observation;
determining a greatest of the determined candidate estimated returns;
determining a terminal reward associated with a triggering of a failure condition corresponding to a failure of a physical component of the UAV following the UAV visiting the second location of the UAV;
determining, using the determined terminal reward and the greatest candidate estimated return, a second estimated return for the first flight action following the first observation, accounting for an intervention by an adversarial stopping agent arranged to trigger the failure condition when predetermined stopping criteria are satisfied; and
updating the current parameter values of the value estimator in dependence upon a difference between the first estimated return and the second estimated return,
wherein, after being sequentially updated in accordance with each of the one or more experience tuples, the current parameter values of the value estimator are trained parameter values.

### Claim 2 (depends on 1)

The method of claim 1, wherein the terminal reward is determined in dependence on location data indicating the location of the UAV with respect to a predetermined map when the failure condition is triggered.

### Claim 3

A computer-implemented method comprising:
receiving data indicative of one or more experience tuples each comprising a first observation characterizing a first state of an environment, a first action performed by a control agent in dependence on the first observation, a reward associated with the performance of the first action, and a second observation characterizing a second state of the environment following the performance of the first action;
for each of the one or more experience tuples:
processing the first observation, using a value estimator with current parameter values, to determine a first estimated return for the first action following the first observation;
processing the second observation, using a target value estimator with an identical architecture to the value estimator, to determine a set of candidate estimated returns, each of the set of candidate estimated returns corresponding to a respective one of a set of candidate second actions following the second observation;
determining a greatest of the set of candidate estimated returns;
determining a terminal reward associated with a triggering of a failure condition in the second state of the environment;
determining, using the determined terminal reward and the greatest candidate estimated return, a second estimated return for the first action following the first observation, accounting for an intervention by an adversarial stopping agent arranged to trigger the failure condition when predetermined stopping criteria are satisfied; and
updating the current parameter values of the value estimator in dependence upon a difference between the first estimated return and the second estimated return,
wherein, after being sequentially updated in accordance with each of the one or more experience tuples, the current parameter values of the value estimator are trained parameter values.

### Claim 4 (depends on 3)

The method of claim 3, wherein the predetermined criteria for triggering the failure condition include the determined terminal reward being lower than the second estimated return for the first observation and the first action.

### Claim 5 (depends on 3)

The method of claim 3, wherein
the environment is a physical environment; and
for each of the one or more experience tuples, the first and second observations are made using one or more sensors.

### Claim 6 (depends on 5)

The method of claim 5, wherein for each of the one or more experience tuples, the first action is performed using one or more actuators.

### Claim 7 (depends on 3)

The method of claim 3, wherein:
the control agent is arranged to control an autonomous vehicle;
the second observation characterizing a second state of the environment is indicative of a current location of the autonomous vehicle;
the failure condition corresponds to a mechanical failure of a physical component of the autonomous vehicle; and
the terminal reward associated with the triggering of the failure condition in the second state of the environment depends on the indicated current location of the autonomous vehicle.

### Claim 8 (depends on 7)

The method of claim 7, wherein the autonomous vehicle is a UAV.

### Claim 9 (depends on 7)

The method of claim 7, wherein the control agent is arranged to determine a route for the autonomous vehicle.

### Claim 10 (depends on 3)

The method of claim 3, wherein:
the environment is a physical environment;
the control agent is arranged to control a physical entity in the physical environment, the physical entity having a plurality of physical components;
the failure condition corresponds to a failure one of the physical components, resulting in a reduced set of actions being available to the control agent; and
the terminal reward associated the triggering of the failure condition in the second state comprises an estimated return for the second observation taking into account the reduced set of actions available to the control agent.

### Claim 11 (depends on 10)

The method of claim 10, wherein said physical components are power supplies for a machine.

### Claim 12 (depends on 10)

The method of claim 10, wherein said physical components are sensors.

### Claim 13 (depends on 10)

The method of claim 10, wherein said physical components are actuators.

### Claim 14 (depends on 3)

The method of claim 3, wherein the value estimator and the target value estimator are identical.

### Claim 15 (depends on 3)

The method of claim 3, comprising updating parameter values of the target value estimator to match the current parameter values of the value estimator after a predetermined number of updates of the current parameter values of the value estimator.

### Claim 16 (depends on 3)

The method of claim 3, wherein:
the value estimator comprises a deep neural network with a given architecture; and
the target value estimator comprises a deep neural network with the same architecture as the value estimator.

### Claim 17 (depends on 3)

the method of claim 3, wherein:
the value estimator comprises a linear combination of predetermined basis functions; and
the target value estimator comprises a linear combination of the same predetermined basis functions as the value estimator.

### Claim 18 (depends on 3)

The method of claim 3, comprising:
receiving data indicative of a third observation characterizing a third state of the environment;
processing the third observation, using the value estimator with the trained parameter values, to determine a candidate estimated return for the third observation and each of a set of candidate third actions; and
determining a best action as the candidate third action determined to have the greatest candidate estimated return.

### Claim 19 (depends on 18)

The method of claim 18, comprising generating further data indicative of a further experience tuple for further training of the value estimator, wherein generating the further experience tuple comprises:
selecting a third action to be performed by the control agent in dependence on the third observation; and
receiving data indicative of a reward associated with the performance of the third action and a fourth observation characterizing a fourth state of the environment following the performance of the third action,
wherein selecting the third action comprises selecting randomly from the set of candidate third actions with a predetermined probability between zero and one, and otherwise selecting the determined best action.

### Claim 20

A data processing system arranged to:
store data indicative of one or more experience tuples each comprising a first observation characterizing a first state of an environment, a first action performed by the control agent in dependence on the first observation, a reward associated with the performance of the first action, and a second observation characterizing a second state of the environment following the performance of the first action;
for each of the one or more experience tuples:
process the first observation, using a value estimator with current parameter values, to determine a first estimated return for the first action following the first observation;
process the second observation, using a target value estimator with an identical architecture to the value estimator, to determine a set of candidate estimated returns, each of the set of candidate estimated returns corresponding to a respective one of a set of candidate second actions following the second observation;
determine a greatest of the set of candidate estimated returns;
determine a terminal reward associated with a triggering of a failure condition in the second state of the environment;
determine, using the determined terminal reward and the greatest candidate estimated return, a second estimated return for the first action following the first observation, accounting for an intervention by an adversarial stopping agent arranged to trigger the failure condition when predetermined stopping criteria are satisfied; and
update the current parameter values of the value estimator in dependence upon a difference between the first estimated return and the second estimated return,
wherein, after being sequentially updated in accordance with each of the one or more experience tuples, the current parameter values of the value estimator are trained parameter values.
