Selecting reinforcement learning actions using goals and observations

A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for selecting an action to be performed by a reinforcement learning agent that interacts with an environment by receiving observations characterizing a current state of the environment and, in response, performing actions from a predetermined set of actions, the operations comprising:

• receiving an observation characterizing a current state of the environment;
• receiving, from a user or from an external system, an input identifying a target state from a set of target states of the environment that is different from the current state of the environment characterized by the observation, wherein the target state is a state to be reached by the reinforcement learning agent by interacting with the environment;
• generating a goal that corresponds to the target state identified in the input received from the user or from the external system;
• and
• processing the goal and the observation using a universal value function approximator (UVFA) that generalizes over states and goals in order to select an action to be performed by the reinforcement learning agent in order to reach the provided target state in response to the observation based on the observation and the generated goal, comprising:
• processing the observation using an observation neural network to generate a numeric representation of the observation;
• processing the goal using a goal neural network to generate a numeric representation of the goal;
• combining the numeric representation of the observation and the numeric representation of the goal to generate a combined representation of the observation and the goal;
• processing the combined representation of the observation and the goal using an action score neural network to generate a respective score for each action in the predetermined set of actions;
• and
• selecting the action to be performed using the respective scores for the actions in the predetermined set of actions.

One or more non-transitory storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for selecting an action to be performed by a reinforcement learning agent that interacts with an environment by receiving observations characterizing a current state of the environment and, in response, performing actions from a predetermined set of actions, the operations comprising:

• receiving an observation characterizing a current state of the environment;
• receiving, from a user or from an external system, an input identifying a target state from a set of target states of the environment that is different from the current state of the environment characterized by the observation, wherein the target state is a state to be reached by the reinforcement learning agent by interacting with the environment;
• generating a goal that corresponds to the target state identified in the input received from the user or from the external system;
• and
• processing the goal and the observation using a universal value function approximator (UVFA) that generalizes over states and goals in order to select an action to be performed by the reinforcement learning agent in order to reach the provided target state in response to the observation based on the observation and the generated goal, comprising:
• processing the observation using an observation neural network to generate a numeric representation of the observation;
• processing the goal using a goal neural network to generate a numeric representation of the goal;
• combining the numeric representation of the observation and the numeric representation of the goal to generate a combined representation of the observation and the goal;
• processing the combined representation of the observation and the goal using an action score neural network to generate a respective score for each action in the predetermined set of actions;
• and
• selecting the action to be performed using the respective scores for the actions in the predetermined set of actions.