Action selection based on environment observations and textual instructions

A system comprising:

• one or more computers;
• and
• one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for selecting actions to be performed by an agent interacting with an environment, the operations comprising:
• at each of a plurality of time steps:
• receiving a current text string in a natural language that expresses information about a current task being performed by the agent;
• receiving a current observation characterizing a current state of the environment;
• processing an input comprising the current text string and the current observation using a policy neural network to generate an action selection output, the processing comprising:
• combining, by the policy neural network and in accordance with values of a set of policy neural network parameters, the current text string and the current observation to produce a combined embedding;
• and
• generating, by the policy neural network and in accordance with the values of the set of policy neural network parameters, the action selection output based on the combined embedding;
• and
• selecting an action to be performed by the agent at the time step based on the action selection output;
• wherein the policy neural network has been trained from end-to-end using reinforcement learning.

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for selecting actions to be performed by an agent interacting with an environment, the operations comprising:

• at each of a plurality of time steps:
• receiving a current text string in a natural language that expresses information about a current task being performed by the agent;
• receiving a current observation characterizing a current state of the environment;
• processing an input comprising the current text string and the current observation using a policy neural network to generate an action selection output, the processing comprising:
• combining, by the policy neural network and in accordance with values of a set of policy neural network parameters, the current text string and the current observation to produce a combined embedding;
• and
• generating, by the policy neural network and in accordance with the values of the set of policy neural network parameters, the action selection output based on the combined embedding;
• and
• selecting an action to be performed by the agent at the time step based on the action selection output;
• wherein the policy neural network has been trained from end-to-end using reinforcement learning.