Controlling agents using reinforcement learning with mixed-integer programming

A system comprising one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• obtaining a plurality of transitions that are each generated as a result of an agent interacting with an environment, each transition comprising:
• a current observation characterizing a current state of the environment,
• a current action performed by the agent in response to the current observation,
• a reward received in response to the agent performing the current action, and
• a next observation characterizing a next state of the environment;
• training a Q neural network having a plurality of Q network parameters on the transitions, wherein the Q neural network is configured to process an observation and initial action constraints in accordance with the Q network parameters to generate a mixed-integer programming (MIP) problem based on a Q value objective and the initial action constraints, the initial action constraints specifying a set of possible actions that can be performed by the agent to interact with the environment,
• the training comprising, for each of one or more of the plurality of transitions:
• processing the next observation and initial action constraints specifying a set of possible next actions to perform in response to the next observation using the Q neural network in accordance with current values of the Q network parameters to generate the mixed-integer programming (MIP) problem including defining a Q value objective function and a set of action constraints;
• evaluating the MIP problem to identify a next action that achieves the Q value objective and meets the set of action constraints;
• determining, based on the next observation and the next action, a temporal difference learning target for the transition;
• determining, based on processing the current observation and initial action constraints specifying the current action using the Q neural network in accordance with current values of the Q network parameters, a current Q value for the transition;
• determining a temporal difference learning error for the transition by computing a difference between the current Q value and the temporal difference learning target;
• and
• using the temporal difference learning error in determining an update to the current values of the Q network parameters.

One or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• obtaining a plurality of transitions that are each generated as a result of an agent interacting with an environment, each transition comprising:
• a current observation characterizing a current state of the environment,
• a current action performed by the agent in response to the current observation,
• a reward received in response to the agent performing the current action, and
• a next observation characterizing a next state of the environment;
• training a Q neural network having a plurality of Q network parameters on the transitions, wherein the Q neural network is configured to process an observation and initial action constraints in accordance with the Q network parameters to generate a mixed-integer programming (MIP) problem based on a Q value objective and the initial action constraints, the initial action constraints specifying a set of possible actions that can be performed by the agent to interact with the environment,
• the training comprising, for each of one or more of the plurality of transitions:
• processing the next observation and initial action constraints specifying a set of possible next actions to perform in response to the next observation using the Q neural network in accordance with current values of the Q network parameters to generate the mixed-integer programming (MIP) problem including defining a Q value objective function and a set of action constraints;
• evaluating the MIP problem to identify a next action that achieves the Q value objective and meets the set of action constraints;
• determining, based on the next observation and the next action, a temporal difference learning target for the transition;
• determining, based on processing the current observation and initial action constraints specifying the current action using the Q neural network in accordance with current values of the Q network parameters, a current Q value for the transition;
• determining a temporal difference learning error for the transition by computing a difference between the current Q value and the temporal difference learning target;
• and
• using the temporal difference learning error in determining an update to the current values of the Q network parameters.