# US 2021/0341885 A1 · Device and method for training a control strategy for a control device over several iterations

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0341885 A1
- Document ID: US20210341885A1
- Status: Published application
- Kind: A1
- Application number: 17/191,091
- Filed: 2021-03-03
- Published: 2021-11-04
- Earliest claimed: 2020-04-30
- Assignees: Robert Bosch GmbH
- Applicants: Robert Bosch GmbH
- Inventors: Felix Schmitt; Johannes Maximilian Doellinger
- Web page: https://patentgrep.com/patent/US20210341885A1

## CPC

- G05B 13/027: {using neural networks only}
- G05B 13/042: {in which a parameter or coefficient is automatically adjusted to optimise the performance}
- G05B 17/02: electric
- G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00)
- G05D 1/0221: {involving a learning process}
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/08: Learning methods
- G06N 3/092: Reinforcement learning
- G05D 2101/15: using machine learning, e.g. neural networks
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method of training a control strategy for a control. An exploration strategy for a current version of the control strategy is determined in each of several iterations. Several simulation runs are carried out, in each of which an action is selected in accordance with the exploration strategy, and it being checked if the selected action is safe, until a safe action has been selected or a maximum number of actions greater than or equal to two has been selected. A follow-up state of the state in the sequence of states is ascertained. The sequence of states are collected as data of the simulation run; for the iteration. The value of a loss function is ascertained over the data of the executed simulation runs and the control strategy is adapted so that the value of the loss function is reduced.

## Claims

### Claim 1

A method for training a control strategy for a control device over several iterations, the method comprising, in each iteration:
determining an exploration strategy for a current version of the control strategy;
carrying out several simulation runs, including, for each of the simulation runs, performing:
for each state of a sequence of states beginning with an initial state of the simulation run:
selecting an action in accordance with the exploration strategy, and checking if the selected action is safe, until a safe action has been selected or a maximum number of actions greater than or equal to two has been selected, and
(i) ascertaining a follow-up state of the state in the sequence of states by simulation during execution of the selected action, if a safe action has been selected, or (ii) if no safe action has been selected up to reaching a maximum number in accordance with the exploration strategy, interrupting the simulation run is interrupted, or selecting a specified, safe action is selected, if one is available and the follow-up state of the state in the sequence of states is ascertained by simulation during the execution of the selected, predefined, safe action, and
collecting the sequence of states, including the selected actions and rewards received in the states, as data of the simulation run;
ascertaining a value of a loss function over the data of the executed simulation runs for the iteration; and
adapting the control strategy to a new version, so that the value of the loss function is reduced.

### Claim 2 (depends on 1)

The method as recited in claim 1, wherein the loss function includes an auxiliary term, which is a function of the data of the executed simulation runs, and which penalizes selection of unsafe actions in the simulation runs.

### Claim 3 (depends on 2)

The method as recited in claim 2, wherein the auxiliary term is weighted, and beginning with a starting weight, the weighting is reduced in the course of the iterations.

### Claim 4 (depends on 1)

The method as recited in claim 1, wherein the control strategy is given by a neural network, and the adaptation of the control strategy includes an adaptation of weightings of the neural network.

### Claim 5 (depends on 1)

The method as recited in claim 1, wherein each simulation run is ended, when the sequence of states reaches a maximum length, or when a control target is reached in the ascertained follow-up state.

### Claim 6 (depends on 1)

The method as recited in claim 1, wherein the loss function is a loss function for an actor-critic RL method or a deep Q-learning method.

### Claim 7 (depends on 1)

The method as recited in claim 1, wherein beginning with an initial value, the maximum number is reduced in the course of the iterations.

### Claim 8

A device configured to train a control strategy for a control device over several iterations, the device configured to, in each iteration:
determine an exploration strategy for a current version of the control strategy;
carry out several simulation runs, for each of the simulation runs, the device is configured to:
for each state of a sequence of states beginning with an initial state of the simulation run:
select an action in accordance with the exploration strategy, and checking if the selected action is safe, until a safe action has been selected or a maximum number of actions greater than or equal to two has been selected, and
(i) ascertain a follow-up state of the state in the sequence of states by simulation during execution of the selected action, if a safe action has been selected, or (ii) if no safe action has been selected up to reaching a maximum number in accordance with the exploration strategy, interrupt the simulation run, or select a specified, safe action, if one is available and the follow-up state of the state in the sequence of states is ascertained by simulation during the execution of the selected, predefined, safe action, and
collect the sequence of states, including the selected actions and rewards received in the states, as data of the simulation run;
ascertain a value of a loss function over the data of the executed simulation runs for the iteration; and
adapt the control strategy to a new version, so that the value of the loss function is reduced.

### Claim 9

A control method, comprising:
training a control strategy for a control device over several iterations, including, in each iteration:
determining an exploration strategy for a current version of the control strategy;
carrying out several simulation runs, for each of the simulation runs, performing:
for each state of a sequence of states beginning with an initial state of the simulation run:
selecting an action in accordance with the exploration strategy, and checking if the selected action is safe, until a safe action has been selected or a maximum number of actions greater than or equal to two has been selected, and
(i) ascertaining a follow-up state of the state in the sequence of states by simulation during execution of the selected action, if a safe action has been selected, or (ii) if no safe action has been selected up to reaching a maximum number in accordance with the exploration strategy, interrupting the simulation run is interrupted, or selecting a specified, safe action is selected, if one is available and the follow-up state of the state in the sequence of states is ascertained by simulation during the execution of the selected, predefined, safe action, and collecting the sequence of states, including the selected actions and rewards received in the states, as data of the simulation run;
ascertaining a value of a loss function over the data of the executed simulation runs for the iteration; and
adapting the control strategy to a new version, so that the value of the loss function is reduced; and
controlling a software or hardware agent based on the trained control strategy.

### Claim 10

A software or hardware agent, comprising:
a sensor configured to provide sensor data; and
a control device configured to control components of the software or hardware agent based on a trained control strategy, the control strategy being trained over several iterations, in each iteration:
an exploration strategy is determined for a current version of the control strategy;
several simulation runs are carried out, for each of the simulation runs:
for each state of a sequence of states beginning with an initial state of the simulation run:
an action is selected in accordance with the exploration strategy, and it is checked if the selected action is safe, until a safe action has been selected or a maximum number of actions greater than or equal to two has been selected, and
(i) a follow-up state of the state in the sequence of states is ascertained by simulation during execution of the selected action, if a safe action has been selected, or (ii) if no safe action has been selected up to reaching a maximum number in accordance with the exploration strategy, the simulation run is interrupted, or a specified, safe action is selected, if one is available and the follow-up state of the state in the sequence of states is ascertained by simulation during the execution of the selected, predefined, safe action, and
the sequence of states is collected, including the selected actions and rewards received in the states, as data of the simulation run;
a value of a loss function over the data of the executed simulation runs for the iteration is ascertained; and
the control strategy is adapted to a new version, so that the value of the loss function is reduced.

### Claim 11 (depends on 10)

The software or hardware agent as recited in claim 10, wherein the software or hardware agent is a robot.

### Claim 12 (depends on 10)

The software or hardware agent as recited in claim 10, further comprising:
at least one actuator;
wherein the control device is configured to control the at least one actuator in accordance with the trained control strategy.

### Claim 13

A non-transitory computer-readable storage medium on which is stored a computer program for training a control strategy for a control device over several iterations, the computer program, when executed by one or more processors, causing the one or more processors to perform, in each iteration:
determining an exploration strategy for a current version of the control strategy;
carrying out several simulation runs, including, for each of the simulation runs, performing:
for each state of a sequence of states beginning with an initial state of the simulation run:
selecting an action in accordance with the exploration strategy, and checking if the selected action is safe, until a safe action has been selected or a maximum number of actions greater than or equal to two has been selected, and
(i) ascertaining a follow-up state of the state in the sequence of states by simulation during execution of the selected action, if a safe action has been selected, or (ii) if no safe action has been selected up to reaching a maximum number in accordance with the exploration strategy, interrupting the simulation run is interrupted, or selecting a specified, safe action is selected, if one is available and the follow-up state of the state in the sequence of states is ascertained by simulation during the execution of the selected, predefined, safe action, and
collecting the sequence of states, including the selected actions and rewards received in the states, as data of the simulation run;
ascertaining a value of a loss function over the data of the executed simulation runs for the iteration; and
adapting the control strategy to a new version, so that the value of the loss function is reduced.