Selecting actions from large discrete action sets using reinforcement learning

A system comprising one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• receiving a particular observation representing a particular state of an environment;
• and
• selecting an action from a discrete set of actions to be performed by an agent interacting with the environment, wherein each action in the discrete set of actions is represented by a respective point in a multi-dimensional space, and wherein selecting the action comprises:
• processing the particular observation using an actor policy network having a plurality of parameters, wherein the actor policy network is a neural network that is configured to receive the particular observation and to process the observation to generate an ideal point in the multi-dimensional space in accordance with first values of the parameters of the actor policy network, wherein the ideal point does not represent any of the actions in the discrete set of actions;
• determining, from the points that represent actions in the discrete set of actions, the k nearest points to the ideal point, wherein k is an integer greater than one;
• for each nearest point of the k nearest points:
• processing the nearest point and the particular observation using a Q network to generate a respective Q value for the action represented by the nearest point, wherein the Q network is a neural network that has a plurality of parameters and that is configured to receive the particular observation and the nearest point and to process the particular observation and the nearest point to generate the respective Q value for the action represented by the nearest point in accordance with first values of the parameters of the Q network;
• and
• selecting, as the action to be performed by the agent, the action from the k actions represented by the k nearest points that has a highest respective Q value among the k actions.

One or more non-transitory computer storage media storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

• receiving a particular observation representing a particular state of an environment;
• and
• selecting an action from a discrete set of actions to be performed by an agent interacting with the environment, wherein each action in the discrete set of actions is represented by a respective point in a multi-dimensional space, and wherein selecting the action comprises:
• processing the particular observation using an actor policy network having a plurality of parameters, wherein the actor policy network is a neural network that is configured to receive the particular observation and to process the observation to generate an ideal point in the multi-dimensional space in accordance with first values of the parameters of the actor policy network, wherein the ideal point does not represent any of the actions in the discrete set of actions;
• determining, from the points that represent actions in the discrete set of actions, the k nearest points to the ideal point, wherein k is an integer greater than one;
• for each nearest point of the k nearest points:
• processing the nearest point and the particular observation using a Q network to generate a respective Q value for the action represented by the nearest point, wherein the Q network is a neural network that has a plurality of parameters and that is configured to receive the particular observation and the nearest point and to process the particular observation and the nearest point to generate the respective Q value for the action represented by the nearest point in accordance with first values of the parameters of the Q network;
• and
• selecting, as the action to be performed by the agent, the action from the k actions represented by the k nearest points that has a highest respective Q value among the k actions.