Training action selection neural networks using apprenticeship

A computer-implemented system for controlling an agent interacting with an environment to perform a task, comprising:

• one or more processors;
• and
• one or more storage devices storing instructions that when executed by the one or more computers, cause the one or more processors to perform operations comprising:
• receiving an observation comprising input state data characterizing a state of the environment;
• processing the input state data using an actor neural network to generate an output including output action data defining an output action;
• and
• controlling the agent to perform the output action;
• wherein the actor neural network has been trained jointly with a critic neural network that is configured to define a value function that generates an error signal based on an input comprising action data defining an action, state data characterizing the state of the environment, and return data derived from reward data representing a reward from the action performed in a training process;
• wherein the training process has been performed based on training data including data from a demonstration of the task being performed by an expert agent within the environment, wherein the data from the demonstration includes demonstration transition data for a series of demonstration transitions including demonstration examples of the state data, the action data, the reward data, and new state data representing a new state that have been generated as a result of the expert agent interacting with the environment;
• wherein during the training process, the actor neural network has been used to operate on the environment to generate operation transition data comprising operational examples of the state data, the action data, the reward data and the new state data, and the actor neural network and the critic neural network have been trained off-policy using the error signal and using stored tuples sampled from a replay buffer comprising tuples from both the operation transition data and the demonstration transition data.

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for controlling an agent interacting with an environment to perform a task, the operations comprising:

• receiving an observation comprising input state data characterizing a state of the environment;
• processing the input state data using an actor neural network to generate an output including output action data defining an output action;
• and
• controlling the agent to perform the output action;
• wherein the actor neural network has been trained jointly with a critic neural network that is configured to define a value function that generates an error signal based on an input comprising action data defining an action, state data characterizing the state of the environment, and return data derived from reward data representing a reward from the action performed in a training process;
• wherein the training process has been performed based on training data including data from a demonstration of the task being performed by an expert agent within the environment, wherein the data from the demonstration includes demonstration transition data for a series of demonstration transitions including demonstration examples of the state data, the action data, the reward data, and new state data representing a new state that have been generated as a result of the expert agent interacting with the environment;
• wherein during the training process, the actor neural network has been used to operate on the environment to generate operation transition data comprising operational examples of the state data, the action data, the reward data and the new state data, and the actor neural network and the critic neural network have been trained off-policy using the error signal and using stored tuples sampled from a replay buffer comprising tuples from both the operation transition data and the demonstration transition data.