Methods and systems for reinforcement learning

A method for training an agent, the method comprising:

• building a model of an environment using historical information about the environment, wherein the model references a plurality of dynamics of the environment as the agent interacts with the environment, each dynamic comprising:
• a current state of the environment,
• an action to be performed,
• a value associated with the action, and
• a next state of the environment to which the current state transitions after taking the action;
• training an initial policy using an offline process acting on the model of the environment, the initial policy represented as a deep neural network (DNN) defined by a plurality of policy weights that prioritize between available actions to identify a selected action;
• incorporating the initial policy into an online process acting in a first live environment, the online process being free of a model that simulates the plurality of dynamics of the environment, the first live environment defined by an online stream of information describing present dynamics of the first live environment, wherein the online process is a deep concurrent temporal difference (DCTD) algorithm that:
• observes a state of the first live environment;
• identifies a plurality of possible actions to be taken in a first live environment;
• selects an action from among the plurality of possible actions, the action being selected by applying the DNN with the online process to evaluate a value over time resulting from the actions in view of the present dynamics of the first live environment in view of the policy weights;
• observes a value associated with the selected action;
• retrains the online process based on the observed value by updating the policy weights defining the DNN in a direction of reducing a temporal difference error associated with the DNN;
• and
• applying the retrained online process in a second live environment defined by a different online stream of information defining different dynamics as compared to the first live environment, where the online process acts concurrently in the first live environment and the second live environment.

An apparatus configured to train an agent implemented at least partially in hardware, the apparatus comprising:

• a non-transitory device-readable medium storing a model of an environment using historical information about the environment, wherein the model references a plurality of dynamics of the environment as the agent interacts with the environment, each dynamic comprising:
• a current state of the environment,
• an action to be performed,
• a value associated with the action, and
• a next state of the environment to which the current state transitions after taking the action;
• a hardware processor circuit;
• offline training logic implemented at least partially in hardware and configured to train an initial policy using an offline process acting on the model of the environment, the initial policy represented as a deep neural network (DNN) defined by a plurality of policy weights that prioritize between available actions to identify a selected action;
• online training logic implemented at least partially in hardware and configured to:
• incorporate the initial policy into an online process acting in a first live environment, the online process being free of a model that simulates the plurality of dynamics of the environment, the first live environment defined by an online stream of information describing present dynamics of the first live environment, wherein the online process is a deep concurrent temporal difference (DCTD) algorithm that:
• observes a state of the first live environment;
• identifies a plurality of possible actions to be taken in a first live environment,
• selects the online process to select an action from among the plurality of possible actions, the action being selected by applying the DNN with the online process to evaluate a value over time resulting from the actions in view of the present dynamics of the first live environment in view of the policy weights,
• observes a value associated with the selected action, and
• retrains the online process based on the observed value by updating the policy weights defining the DNN in a direction of reducing a temporal difference error associated with the DNN;
• and
• application logic implemented at least partially in hardware and configured to apply the retrained online process in a second live environment defined by a different online stream of information defining different dynamics as compared to the first live environment, where the online process acts concurrently in the first live environment and the second live environment.