# US 2021/0383245 A1 · Device and method for planning an operation of a technical system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0383245 A1
- Document ID: US20210383245A1
- Status: Published application
- Kind: A1
- Application number: 17/242,790
- Filed: 2021-04-28
- Published: 2021-12-09
- Earliest claimed: 2020-06-05
- Applicants: Robert Bosch GmbH
- Inventors: Jonathan Spitz; Andre Biedenkapp; David Speck; Frank Hutter; Marius Lindauer; Robert Mattmueller
- Web page: https://patentgrep.com/patent/US20210383245A1

## CPC

- G06N 3/08: Learning methods
- G05B 19/045: using logic state machines, consisting only of a memory or a programmable logic device containing the logic for the controlled machine and in which the state of its outputs is dependent on the state of its inputs or part of its own output states, e.g. binary decision controllers, finite state controllers
- G05B 19/056: {Programming the PLC}
- G06F 18/2178: {based on feedback of a supervisor}
- G06N 3/0499: Feedforward networks
- G06N 3/092: Reinforcement learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G05B 2219/13009: State machine instructions
- G06N 3/008: based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A computer-implemented method for planning an operation of a technical system within its environment. The method includes: obtaining state information comprising: a current domain, a time step and a current state; determining by heuristics costs for reachable states from the current state; selecting a heuristics by a policy out of a set of predefined heuristics depending on the state information and costs; choosing the state with the lowest cost returned by the selected heuristic from the reachable states, and determining an operation of the technical system out of the set of possible operation that has to be carried out by the technical system to reach said state with the lowest costreturned by the selected heuristic.

## Claims

### Claim 1

A computer-implemented method for planning an operation of a technical system within an environment of the technical system, the environment being characterized by a current domain out of a set of different respective domains, a current state out of a set of states of the respective domains, and a set of possible operations which can be carried out by the technical system, the method comprising the following steps:
i) obtaining state information including at least the current domain, a time step, and the current state of the environment;
ii) determining, by each heuristic out of a set of predefined heuristics, costs for a plurality of reachable states from the current state, wherein the heuristics are configured to estimate costs to reach a goal state from a given state;
iii) selecting a heuristic out of the set of predefined heuristics by a policy depending on the state information, wherein the policy has been trained to select the heuristic from the set of predefined heuristics, such that a minimal number of state expansions is expected when planning a path to the goal state;
iv) choosing a state with a lowest cost determined by the selected heuristic by the policy from the reachable states; and
v) determining an operation of the technical system out of the set of possible operation that has to be carried out by the technical system to reach the state with the lowest cost determined by the selected heuristic.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the current state for each domain out of the plurality of domains is characterized by at least the following features: maximum cost that can be returned by each heuristic of the set of predefined heuristics, minimum cost that can be returned by each heuristic of the set of predefined heuristics, average costs returned from each heuristic of the set of predefined heuristics, variance of costs returned from each heuristic of the set of predefined heuristics, number of states maintained by each heuristic of the set of predefined heuristics, and a current time step.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the state further includes a features reflecting context information of the current domain.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the steps i) to iv) are subsequently carried out several times until the current state corresponds to the goal state, wherein the chosen states with the lowest costs are stored in a list, wherein depending on the list, a sequence of operations is determined which generates a sequence of states of the list to reach the goal state.

### Claim 5 (depends on 4)

The method according to claim 4, wherein, for each heuristic, a list is used and a most promising state with a lowest cost of the corresponding list of the selected heuristic by the policy is expanded.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the set of heuristics includes at least one of the following heuristics: fast-forward planning heuristic or causal graph heuristic or context-enhanced additive heuristic or an additive heuristic.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the policy is trained via reinforcement learning.

### Claim 8 (depends on 7)

The method according to claim 7, wherein the policy is trained by Dynamic Algorithm Control (DAC).

### Claim 9 (depends on 8)

The method according to claim 8, wherein a sparse reward function is utilized.

### Claim 10 (depends on 1)

The method according to claim 1, wherein the technical system is a robot or a transportation system, wherein the operations corresponds to predefined movements of the robot or the transportation system.

### Claim 11

A non-transitory machine-readable storage medium on which is stored a computer program for planning an operation of a technical system within an environment of the technical system, the environment being characterized by a current domain out of a set of different respective domains, a current state out of a set of states of the respective domains, and a set of possible operations which can be carried out by the technical system, the computer program, when executed by a computer, causing the computer to perform the following steps:
i) obtaining state information including at least the current domain, a time step, and the current state of the environment;
ii) determining, by each heuristic out of a set of predefined heuristics, costs for a plurality of reachable states from the current state, wherein the heuristics are configured to estimate costs to reach a goal state from a given state;
iii) selecting a heuristic out of the set of predefined heuristics by a policy depending on the state information, wherein the policy has been trained to select the heuristic from the set of predefined heuristics, such that a minimal number of state expansions is expected when planning a path to the goal state;
iv) choosing a state with a lowest cost determined by the selected heuristic by the policy from the reachable states; and
v) determining an operation of the technical system out of the set of possible operation that has to be carried out by the technical system to reach the state with the lowest cost determined by the selected heuristic.

### Claim 12

A system for planning an operation of a technical system within an environment of the technical system, the environment being characterized by a current domain out of a set of different respective domains, a current state out of a set of states of the respective domains, and a set of possible operations which can be carried out by the technical system, the system configured to:
i) obtain state information including at least the current domain, a time step, and the current state of the environment;
ii) determine, by each heuristic out of a set of predefined heuristics, costs for a plurality of reachable states from the current state, wherein the heuristics are configured to estimate costs to reach a goal state from a given state;
iii) select a heuristic out of the set of predefined heuristics by a policy depending on the state information, wherein the policy has been trained to select the heuristic from the set of predefined heuristics, such that a minimal number of state expansions is expected when planning a path to the goal state;
iv) choose a state with a lowest cost determined by the selected heuristic by the policy from the reachable states; and
v) determine an operation of the technical system out of the set of possible operation that has to be carried out by the technical system to reach the state with the lowest cost determined by the selected heuristic.
