Reinforcement learning using density estimation with online clustering for exploration

One or more computer storage media storing instructions that when executed by one or more computers cause the one or more computers to operations for training a computer-implemented neural network system used to control an agent interacting with an environment, wherein the computer-implemented neural network system comprises:

• an action selection neural network system used to select actions to be performed by the agent, a representation generation neural network system configured to process an observation of a state of the environment to generate a state embedding, and episodic memory;
• the operations comprising, at a plurality of time steps:
• obtaining a current observation characterizing a current state of the environment;
• processing the current observation using the action selection neural network system to select an action to be performed by the agent at the time step;
• obtaining a subsequent observation characterizing a subsequent state of the environment after the agent performs the selected action;
• processing the subsequent observation using the representation generation neural network system to generate a subsequent state embedding representing at least the subsequent observation;
• and
• performing an update step on data stored in the episodic memory using the subsequent state embedding;
• the method further comprising:
• maintaining in the episodic memory a record of previously visited regions of a state space of the environment as data comprising state embedding cluster centers representing centers of the previously visited regions in a space of the state embeddings, and for each state embedding cluster center a corresponding count value representing a visitation count of the corresponding region of state space of the environment by the agent;
• generating an intrinsic reward, using the state embedding cluster center and corresponding count value data stored in the episodic memory, to reward exploration of states of the environment by the agent;
• and
• training the action selection neural network system based on a reward including the intrinsic reward;
• and wherein performing the update step comprises:
• updating or replacing one or more of the state embedding cluster centers and one or more of the corresponding count values using the subsequent state embedding.

A system comprising:

• one or more computers;
• and
• one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training a computer-implemented neural network system used to control an agent interacting with an environment, wherein the computer-implemented neural network system comprises:
• an action selection neural network system used to select actions to be performed by the agent, a representation generation neural network system configured to process an observation of a state of the environment to generate a state embedding, and episodic memory;
• the operations comprising, at a plurality of time steps:
• obtaining a current observation characterizing a current state of the environment;
• processing the current observation using the action selection neural network system to select an action to be performed by the agent at the time step;
• obtaining a subsequent observation characterizing a subsequent state of the environment after the agent performs the selected action;
• processing the subsequent observation using the representation generation neural network system to generate a subsequent state embedding representing at least the subsequent observation;
• and
• performing an update step on data stored in the episodic memory using the subsequent state embedding;
• the method further comprising:
• maintaining in the episodic memory a record of previously visited regions of a state space of the environment as data comprising state embedding cluster centers representing centers of the previously visited regions in a space of the state embeddings, and for each state embedding cluster center a corresponding count value representing a visitation count of the corresponding region of state space of the environment by the agent;
• generating an intrinsic reward, using the state embedding cluster center and corresponding count value data stored in the episodic memory, to reward exploration of states of the environment by the agent;
• and
• training the action selection neural network system based on a reward including the intrinsic reward;
• and wherein performing the update step comprises:
• updating or replacing one or more of the state embedding cluster centers and one or more of the corresponding count values using the subsequent state embedding.