Dynamic policy programming for continuous action spaces

A system, comprising:

• one or more processors;
• and
• a computer-readable storage medium, coupled to the one or more processors, storing program instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
• generating a replay buffer including a set of transitions from a set of states, a set of actions based on a current policy, a set of subsequent states based on the set of actions, and a set of rewards based on the set of actions;
• sampling a plurality of transitions from the replay buffer as a set of samples;
• generating an action value based on a dynamic policy programming (DPP) recursion using the set of samples and a first hyperparameter;
• evaluating the policy by computing a KL divergence using the set of samples, the action value and a second hyperparameter;
• and
• generating a subsequent policy by updating the current policy by minimizing a sum of two KL divergence and using a third hyperparameter.

A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions being executable by one or more processors to cause the one or more processors to perform operations comprising:

• generating a replay buffer including a set of transitions from a set of states, a set of actions based on a current policy, a set of subsequent states based on the set of actions, and a set of rewards based on the set of actions;
• sampling a plurality of transitions from the replay buffer as a set of samples;
• generating an action value based on a dynamic policy programming (DPP) recursion using the set of samples and a first hyperparameter;
• evaluating the policy by computing a KL divergence using the set of samples, the action value and a second hyperparameter;
• and
• generating a subsequent policy by updating the current policy by minimizing a sum of two KL divergence and using a third hyperparameter.