Imitation learning using a generative predecessor neural network

A system comprising:

• one or more computers;
• and
• one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training an action selection policy neural network, wherein the action selection policy neural network has a plurality of action selection policy neural network parameters, wherein the action selection policy neural network is configured to process an observation characterizing a state of an environment in accordance with values of the action selection policy neural network parameters to generate an action selection policy output, wherein the action selection policy output comprises a respective action score for each action in a predetermined set of possible actions, wherein the action selection policy output is used to select an action to be performed by an agent interacting with an environment, the operations comprising:
• obtaining an expert observation which characterizes a state of the environment that has been classified as being relevant to accomplishing a particular task;
• processing the expert observation using a generative neural network system to generate a predecessor observation - predecessor action pair,
• wherein the predecessor observation - predecessor action pair comprises:
• (i) a predecessor observation characterizing a state of the environment, and (ii) a predecessor action from the predetermined set of possible actions,
• wherein performing the predecessor action in response to the predecessor observation is predicted to cause the environment to subsequently transition into the state characterized by the expert observation after one or more time steps;
• processing the predecessor observation using the action selection policy neural network in accordance with current values of the action selection policy neural network parameters to generate an action score for the predecessor action;
• and
• adjusting the current values of the action selection policy neural network parameters to increase the action score for the predecessor action which is generated by the action selection policy neural network by processing the predecessor observation.

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training an action selection policy neural network, wherein the action selection policy neural network has a plurality of action selection policy neural network parameters, wherein the action selection policy neural network is configured to process an observation characterizing a state of an environment in accordance with values of the action selection policy neural network parameters to generate an action selection policy output, wherein the action selection policy output comprises a respective action score for each action in a predetermined set of possible actions, wherein the action selection policy output is used to select an action to be performed by an agent interacting with an environment, the operations comprising:

• obtaining an expert observation which characterizes a state of the environment that has been classified as being relevant to accomplishing a particular task;
• processing the expert observation using a generative neural network system to generate a predecessor observation - predecessor action pair,
• wherein the predecessor observation - predecessor action pair comprises:
• (i) a predecessor observation characterizing a state of the environment, and (ii) a predecessor action from the predetermined set of possible actions,
• wherein performing the predecessor action in response to the predecessor observation is predicted to cause the environment to subsequently transition into the state characterized by the expert observation after one or more time steps;
• processing the predecessor observation using the action selection policy neural network in accordance with current values of the action selection policy neural network parameters to generate an action score for the given predecessor action;
• and
• adjusting the current values of the action selection policy neural network parameters to increase the action score for the predecessor action which is generated by the action selection policy neural network by processing the predecessor observation.