Systems and methods for risk-sensitive reinforcement learning
AssigneeJPMorgan Chase Bank, N.A.
InventorsNelson VADORI, Sumitra GANESH, Maria Manuela VELOSO
Life of this patent
Estimating the term…
Abstract
Systems and methods for risk-sensitive reinforcement learning are disclosed. In one embodiment, a method for a method for training a risk-sensitive reinforcement learning policy may include: (1) receiving, from a data source, a plurality of sets of training data for a plurality of time steps; (2) receiving a training budget comprising a plurality of episodes, a risk aversion coefficient, and an end state; and (3) calculating a correction factor using the training data. The correction factor may minimize stochasticity based on a risk aversion coefficient.
Claims18
A method for a method for training a risk-sensitive reinforcement learning policy, comprising:
- a risk-sensitive learning engine comprising at least one computer processor receiving, from a data source, a plurality of sets of training data for a plurality of time steps, each set of training data comprising an initial state, an action, a reward, and a state at the next time step;
- the risk-sensitive learning engine receiving a training budget comprising a plurality of episodes, a risk aversion coefficient, and an end state;
- and
- for each episode in the episode in the training budget:
- the risk-sensitive learning engine initializing a Q table;
- the risk-sensitive learning engine executing an action for time t=0 and monitoring a reward at t+1 and a state at time t+1;
- the risk-sensitive learning engine increasing time t to t+1;
- the risk-sensitive learning engine executing an action for time t and monitoring a reward at time t+1 and a state at time t+1;
- the risk-sensitive learning engine calculating an average reward over time t;
- the risk-sensitive learning engine calculating a correction factor based on the reward at time t+1, the average reward over time, and the risk aversion coefficient, wherein the correction factor minimizes stochasticity for the reward based on the risk aversion coefficient;
- and
- repeating the steps of increasing time t to t+1, executing an action for time+1 and monitoring the reward at time t+1 and a state at time t+1, calculating the average reward over time t, and calculating a correction factor based on the reward at time t+1 minus and the risk aversion coefficient until the end state is met.
A method for a method for training a risk-sensitive reinforcement learning policy, comprising:
- a risk-sensitive learning engine comprising at least one computer processor calculating, from an initial distribution parameter, an experience replay table, a number of distribution gradient steps, and a number of Stochastic Gradient Descent (SGD) samples, calculating an approximation of an optimal distribution parameter;
- the risk-sensitive learning engine receiving, from a data source, a plurality of sets of training data for a plurality of time steps, each set of training data comprising an initial state, an action, a reward, and a state at the next time step;
- the risk-sensitive learning engine receiving a risk aversion coefficient;
- and
- for each time step t:
- the risk-sensitive learning engine calculating, for a training budget comprising a number of episodes b, a parametric approximation of an average reward over time for the training data;
- the risk-sensitive learning engine calculating a correction factor based on the reward at time t+1, the parametric approximation of the average reward over time, and the risk aversion coefficient, wherein the correction factor minimizes stochasticity for the reward based on the risk aversion coefficient;
- and
- increasing time t to t+1, executing an action for time+1 and monitoring the reward at time t+1 and a state at time t+1, calculating the parametric approximation of an average reward over time t, and calculating the correction factor based on the reward at time t+1 minus and the risk aversion coefficient until an end state is met.
Dependencies are inferred from the claim text.