Method for controlling a robot device and robot device controller

A robot controller configured to control a robot device, the robot controller configured to:

• obtain demonstrations for controlling the robot device;
• perform an initial training of an actor neural network by imitation learning of the demonstrations;
• control the robot device by the initially trained actor neural network to generate multiple trajectories of the robot device, wherein each of the trajectories includes a sequence of actions selected by the initially trained actor neural network in a sequence of states, and observing a return for each of the actions selected by the initially trained actor neural network;
• perform an initial training of a critic neural network by supervised learning, wherein the critic neural network is trained to determine the observed returns of the actions selected by the initially trained actor neural network;
• train the actor neural network and the critic neural network by reinforcement learning starting from the initially trained actor neural network and the initially trained critic neural network;
• and
• control the robot device by the trained actor neural network and trained critic neural network;
• wherein the training of the actor neural network and the critic neural network by reinforcement learning includes performing a plurality of episodes, wherein in each episode of the episodes, the actor neural network selects a control action for each time step of a plurality of time steps and the robot is controlled, for each time step of the plurality of time steps, to perform the selected control action in the time step, wherein the method comprises copying, only after a delay of multiple time steps of the plurality of time steps, the actor neural network to generate a target actor neural network and the critic neural network to generate a target critic neural network, and the method comprises using the target actor neural network to provide, for each control action selected by the actor neural network, one or more control actions following the control action selected by the actor neural network to the target critic neural network, wherein the target critic neural network determines a return of the one or more control actions selected by the target actor neural network, wherein a loss of the critic neural network is determined from a return of the action selected by the actor neural network determined by the critic neural network, a sum of the returns of the one or more control actions selected by the target actor neural network and a return of the action selected by the actor neural network.

A non-transitory computer readable medium on which are stored instructions for controlling a robot device, the instructions, when executed by a processor, causing the processor to perform the following steps:

• obtaining demonstrations for controlling the robot device;
• performing an initial training of an actor neural network by imitation learning of the demonstrations;
• controlling the robot device by the initially trained actor neural network to generate multiple trajectories of the robot device, wherein each of the trajectories includes a sequence of actions selected by the initially trained actor neural network in a sequence of states, and observing a return for each of the actions selected by the initially trained actor neural network;
• performing an initial training of a critic neural network by supervised learning, wherein the critic neural network is trained to determine the observed returns of the actions selected by the initially trained actor neural network;
• training the actor neural network and the critic neural network by reinforcement learning starting from the initially trained actor neural network and the initially trained critic neural network;
• and
• controlling the robot device by the trained actor neural network and trained critic neural network;
• wherein training the actor neural network and the critic neural network by reinforcement learning includes performing a plurality of episodes, wherein in each episode of the episodes, the actor neural network selects a control action for each time step of a plurality of time steps and the robot is controlled, for each time step of the plurality of time steps, to perform the selected control action in the time step, wherein the method comprises copying, only after a delay of multiple time steps of the plurality of time steps, the actor neural network to generate a target actor neural network and the critic neural network to generate a target critic neural network, and the method comprises using the target actor neural network to provide, for each control action selected by the actor neural network, one or more control actions following the control action selected by the actor neural network to the target critic neural network, wherein the target critic neural network determines a return of the one or more control actions selected by the target actor neural network, wherein a loss of the critic neural network is determined from a return of the action selected by the actor neural network determined by the critic neural network, a sum of the returns of the one or more control actions selected by the target actor neural network and a return of the action selected by the actor neural network.