Method for training a control policy for controlling a technical system

A control device configured to train a control policy for controlling a robotic device, the control device comprising a processor and a non-transitory memory storing instructions that, when executed by the processor, cause the control device to:

• receive, by the processor, sensor data from one or more sensors associated with the robotic device, the sensor data representing states of the robotic device interacting with a physical environment;
• train, by the processor, a neural network to implement a value function which, for each state of the robotic device, predicts a cumulative reward that may be obtained by controlling the robotic device, starting from the state, by:
• adapting the neural network for reducing a loss which, for a plurality of states and, for each of the states, for at least one action that has been previously carried out in the state, involves a deviation between a prediction for the cumulative reward by the neural network and an estimation of the cumulative reward that is ascertained from a subsequent state that has been achieved by the action, and a reward that is obtained by the action,
• ascertaining a behavior control policy that reflects a selection of the previously carried out actions in the respective states of the plurality of states,
• wherein in the loss, for each action, the deviation for the action is weighted more strongly the greater a likelihood is that the action is selected by the control policy, in relation to a likelihood that the action is selected by the behavior control policy;
• train the control policy so that it prioritizes actions that result in states for which the neural network predicts a higher value, over actions that result in states for which the neural network predicts a lower value;
• and
• generate, using the trained control policy, a control signal that causes the robotic device to physically perform a task in the physical environment.

A non-transitory computer-readable medium on which is stored a computer program including commands training a control policy for controlling a robotic device, the commands, when executed by a processor, causing the processor to perform the following steps:

• receiving, by the processor, sensor data from one or more sensors associated with the robotic device, the sensor data representing states of the robotic device interacting with a physical environment;
• training, by the processor, a neural network to implement a value function which, for each state of the robotic device, predicts a cumulative reward that may be obtained by controlling the robotic device, starting from the state, by:
• adapting the neural network for reducing a loss which, for a plurality of states and, for each of the states, for at least one action that has been previously carried out in the state, involves a deviation between a prediction for the cumulative reward by the neural network and an estimation of the cumulative reward that is ascertained from a subsequent state that has been achieved by the action, and a reward that is obtained by the action,
• ascertaining a behavior control policy that reflects a selection of the previously carried out actions in the respective states of the plurality of states,
• wherein in the loss, for each action, the deviation for the action is weighted more strongly the greater a likelihood is that the action is selected by the control policy, in relation to a likelihood that the action is selected by the behavior control policy;
• and
• training the control policy so that it prioritizes actions that result in states for which the neural network predicts a higher value, over actions that result in states for which the neural network predicts a lower value;
• and
• generating, using the trained control policy, a control signal that causes the robotic device to physically perform a task in the physical environment.