Sample-efficient reinforcement learning

One or more non-transitory computer readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• maintaining an ensemble of Q networks, each Q network being a neural network that is configured to receive a Q network input comprising (i) an input observation characterizing a state of an environment being interacted with by an agent and (ii) data identifying an action from a set of actions and process the Q network input to generate a Q value for the input observation—action pair;
• maintaining an ensemble of transition models, each transition model being configured to receive a transition input comprising (i) an input observation and (ii) an action performed by the agent in response to the input observation and to process the transition input to generate a predicted next observation characterizing a state that the environment transitioned into as a result of the agent performing the action in response to the observation;
• maintaining an ensemble of reward models, each reward model being configured to receive a reward input comprising (i) an input observation, (ii) an action performed by the agent in response to the input observation, and (iii) a next observation characterizing a state that the environment transitioned into as a result of the agent performing the action in response to the observation and to process the reward input to generate a predicted reward received by the agent in response to performing the action;
• obtaining a transition, the transition comprising (i) an initial training observation, (ii) a training action, (iii) a training reward, and (iv) a next training observation;
• generating, using the ensemble of transition models, M trajectories starting from the next training observation in the transition, each of the M trajectories comprising a respective observation at each of a predetermined number of time steps;
• for each of the M trajectories and for each of the time steps:
• generating, using the ensemble of reward models, N rewards for the time step,
• generating, using the ensemble of Q networks, L Q values for the time step, and
• determining, from the N rewards, the L Q values, and the training reward, L*N candidate target Q values for the trajectory and for the time step;
• for each of the time steps, combining the M*L*N candidate target Q values for the time step from the M trajectories to generate a combined candidate target Q value for the time step;
• determining a final target Q value from the combined target Q value for the time steps;
• and
• training at least one of the Q networks in the ensemble using the final target Q value.

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

• maintaining an ensemble of Q networks, each Q network being a neural network that is configured to receive a Q network input comprising (i) an input observation characterizing a state of an environment being interacted with by an agent and (ii) data identifying an action from a set of actions and process the Q network input to generate a Q value for the input observation—action pair;
• maintaining an ensemble of transition models, each transition model being configured to receive a transition input comprising (i) an input observation and (ii) an action performed by the agent in response to the input observation and to process the transition input to generate a predicted next observation characterizing a state that the environment transitioned into as a result of the agent performing the action in response to the observation;
• maintaining an ensemble of reward models, each reward model being configured to receive a reward input comprising (i) an input observation, (ii) an action performed by the agent in response to the input observation, and (iii) a next observation characterizing a state that the environment transitioned into as a result of the agent performing the action in response to the observation and to process the reward input to generate a predicted reward received by the agent in response to performing the action;
• obtaining a transition, the transition comprising (i) an initial training observation, (ii) a training action, (iii) a training reward, and (iv) a next training observation;
• generating, using the ensemble of transition models, M trajectories starting from the next training observation in the transition, each of the M trajectories comprising a respective observation at each of a predetermined number of time steps;
• for each of the M trajectories and for each of the time steps:
• generating, using the ensemble of reward models, N rewards for the time step,
• generating, using the ensemble of Q networks, L Q values for the time step, and
• determining, from the N rewards, the L Q values, and the training reward, L*N candidate target Q values for the trajectory and for the time step;
• for each of the time steps, combining the M*L*N candidate target Q values for the time step from the M trajectories to generate a combined candidate target Q value for the time step;
• determining a final target Q value from the combined target Q value for the time steps;
• and
• training at least one of the Q networks in the ensemble using the final target Q value.