Off-line learning for robot control using a reward prediction model

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

• obtaining robot experience data characterizing robot interactions with an environment, the robot experience data comprising a plurality of experiences that each comprise (i) an observation characterizing a state of the environment and (ii) an action performed by a respective robot in response to the observation, wherein the experiences comprise:
• expert experiences from episodes of a particular task being performed by an expert agent, and
• unlabeled experiences;
• training, on a first subset of the robot experience data, a reward prediction model that receives a reward input comprising an input observation and generates as output a reward prediction that is a prediction of a task-specific reward for the particular task that should be assigned to the input observation, wherein training the reward prediction model comprises optimizing an objective function that:
• includes a first term that encourages the reward prediction model to assign, to observations from expert experiences, a first reward value that indicates that the particular task was completed successfully after the environment was in the state characterized by the observation, and
• includes a second term that encourages the reward prediction model to assign, to observations from unlabeled experiences, a second reward value that indicates that the particular task was not completed successfully after the environment was in the state characterized by the observation;
• processing experiences in the robot experience data using the trained reward prediction model to generate a respective reward prediction for each of the processed experiences;
• and
• training a policy neural network on (i) the processed experiences and (ii) the respective reward predictions for the processed experiences, wherein the policy neural network is configured to receive a network input comprising an observation and to generate a policy output that defines a control policy for a robot performing the particular task.

One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

• obtaining robot experience data characterizing robot interactions with an environment, the robot experience data comprising a plurality of experiences that each comprise (i) an observation characterizing a state of the environment and (ii) an action performed by a respective robot in response to the observation, wherein the experiences comprise:
• expert experiences from episodes of a particular task being performed by an expert agent, and
• unlabeled experiences;
• training, on a first subset of the robot experience data, a reward prediction model that receives a reward input comprising an input observation and generates as output a reward prediction that is a prediction of a task-specific reward for the particular task that should be assigned to the input observation, wherein training the reward prediction model comprises optimizing an objective function that:
• includes a first term that encourages the reward prediction model to assign, to observations from expert experiences, a first reward value that indicates that the particular task was completed successfully after the environment was in the state characterized by the observation, and
• includes a second term that encourages the reward prediction model to assign, to observations from unlabeled experiences, a second reward value that indicates that the particular task was not completed successfully after the environment was in the state characterized by the observation;
• processing experiences in the robot experience data using the trained reward prediction model to generate a respective reward prediction for each of the processed experiences;
• and
• training a policy neural network on (i) the processed experiences and (ii) the respective reward predictions for the processed experiences, wherein the policy neural network is configured to receive a network input comprising an observation and to generate a policy output that defines a control policy for a robot performing the particular task.