Device and method for training a control strategy for a control device over several iterations

A device configured to train a control strategy for a control device over several iterations, the device configured to, in each iteration:

• determine an exploration strategy for a current version of the control strategy;
• carry out several simulation runs, for each of the simulation runs, the device is configured to:
• for each state of a sequence of states beginning with an initial state of the simulation run:
• select an action in accordance with the exploration strategy, and checking if the selected action is safe, until a safe action has been selected or a maximum number of actions greater than or equal to two has been selected, and
• (i) ascertain a follow-up state of the state in the sequence of states by simulation during execution of the selected action, if a safe action has been selected, or (ii) if no safe action has been selected up to reaching a maximum number in accordance with the exploration strategy, interrupt the simulation run, or select a specified, safe action, if one is available and the follow-up state of the state in the sequence of states is ascertained by simulation during the execution of the selected, predefined, safe action, and
• collect the sequence of states, including the selected actions and rewards received in the states, as data of the simulation run;
• ascertain a value of a loss function over the data of the executed simulation runs for the iteration;
• and
• adapt the control strategy to a new version, so that the value of the loss function is reduced.

A control method, comprising:

• training a control strategy for a control device over several iterations, including, in each iteration:
• determining an exploration strategy for a current version of the control strategy;
• carrying out several simulation runs, for each of the simulation runs, performing:
• for each state of a sequence of states beginning with an initial state of the simulation run:
• selecting an action in accordance with the exploration strategy, and checking if the selected action is safe, until a safe action has been selected or a maximum number of actions greater than or equal to two has been selected, and
• (i) ascertaining a follow-up state of the state in the sequence of states by simulation during execution of the selected action, if a safe action has been selected, or (ii) if no safe action has been selected up to reaching a maximum number in accordance with the exploration strategy, interrupting the simulation run is interrupted, or selecting a specified, safe action is selected, if one is available and the follow-up state of the state in the sequence of states is ascertained by simulation during the execution of the selected, predefined, safe action, and collecting the sequence of states, including the selected actions and rewards received in the states, as data of the simulation run;
• ascertaining a value of a loss function over the data of the executed simulation runs for the iteration;
• and
• adapting the control strategy to a new version, so that the value of the loss function is reduced;
• and
• controlling a software or hardware agent based on the trained control strategy.

A software or hardware agent, comprising:

• a sensor configured to provide sensor data;
• and
• a control device configured to control components of the software or hardware agent based on a trained control strategy, the control strategy being trained over several iterations, in each iteration:
• an exploration strategy is determined for a current version of the control strategy;
• several simulation runs are carried out, for each of the simulation runs:
• for each state of a sequence of states beginning with an initial state of the simulation run:
• an action is selected in accordance with the exploration strategy, and it is checked if the selected action is safe, until a safe action has been selected or a maximum number of actions greater than or equal to two has been selected, and
• (i) a follow-up state of the state in the sequence of states is ascertained by simulation during execution of the selected action, if a safe action has been selected, or (ii) if no safe action has been selected up to reaching a maximum number in accordance with the exploration strategy, the simulation run is interrupted, or a specified, safe action is selected, if one is available and the follow-up state of the state in the sequence of states is ascertained by simulation during the execution of the selected, predefined, safe action, and
• the sequence of states is collected, including the selected actions and rewards received in the states, as data of the simulation run;
• a value of a loss function over the data of the executed simulation runs for the iteration is ascertained;
• and
• the control strategy is adapted to a new version, so that the value of the loss function is reduced.

A non-transitory computer-readable storage medium on which is stored a computer program for training a control strategy for a control device over several iterations, the computer program, when executed by one or more processors, causing the one or more processors to perform, in each iteration:

• determining an exploration strategy for a current version of the control strategy;
• carrying out several simulation runs, including, for each of the simulation runs, performing:
• for each state of a sequence of states beginning with an initial state of the simulation run:
• selecting an action in accordance with the exploration strategy, and checking if the selected action is safe, until a safe action has been selected or a maximum number of actions greater than or equal to two has been selected, and
• (i) ascertaining a follow-up state of the state in the sequence of states by simulation during execution of the selected action, if a safe action has been selected, or (ii) if no safe action has been selected up to reaching a maximum number in accordance with the exploration strategy, interrupting the simulation run is interrupted, or selecting a specified, safe action is selected, if one is available and the follow-up state of the state in the sequence of states is ascertained by simulation during the execution of the selected, predefined, safe action, and
• collecting the sequence of states, including the selected actions and rewards received in the states, as data of the simulation run;
• ascertaining a value of a loss function over the data of the executed simulation runs for the iteration;
• and
• adapting the control strategy to a new version, so that the value of the loss function is reduced.