# US 2021/0232970 A1 · Systems and methods for risk-sensitive reinforcement learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0232970 A1
- Document ID: US20210232970A1
- Status: Published application
- Kind: A1
- Application number: 17/154,825
- Filed: 2021-01-21
- Published: 2021-07-29
- Earliest claimed: 2020-01-24
- Assignees: JPMorgan Chase Bank, N.A.
- Applicants: Jpmorgan Chase Bank, N.A.
- Inventors: Nelson VADORI; Sumitra GANESH; Maria Manuela VELOSO
- Web page: https://patentgrep.com/patent/US20210232970A1

## CPC

- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 20/00: Machine learning
- G06Q 40/06: Asset management; Financial planning or analysis
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Systems and methods for risk-sensitive reinforcement learning are disclosed. In one embodiment, a method for a method for training a risk-sensitive reinforcement learning policy may include: (1) receiving, from a data source, a plurality of sets of training data for a plurality of time steps; (2) receiving a training budget comprising a plurality of episodes, a risk aversion coefficient, and an end state; and (3) calculating a correction factor using the training data. The correction factor may minimize stochasticity based on a risk aversion coefficient.

## Claims

### Claim 1

A method for a method for training a risk-sensitive reinforcement learning policy, comprising:
a risk-sensitive learning engine comprising at least one computer processor receiving, from a data source, a plurality of sets of training data for a plurality of time steps, each set of training data comprising an initial state, an action, a reward, and a state at the next time step;
the risk-sensitive learning engine receiving a training budget comprising a plurality of episodes, a risk aversion coefficient, and an end state; and
for each episode in the episode in the training budget:
the risk-sensitive learning engine initializing a Q table;
the risk-sensitive learning engine executing an action for time t=0 and monitoring a reward at t+1 and a state at time t+1;
the risk-sensitive learning engine increasing time t to t+1;
the risk-sensitive learning engine executing an action for time t and monitoring a reward at time t+1 and a state at time t+1;
the risk-sensitive learning engine calculating an average reward over time t;
the risk-sensitive learning engine calculating a correction factor based on the reward at time t+1, the average reward over time, and the risk aversion coefficient, wherein the correction factor minimizes stochasticity for the reward based on the risk aversion coefficient; and
repeating the steps of increasing time t to t+1, executing an action for time+1 and monitoring the reward at time t+1 and a state at time t+1, calculating the average reward over time t, and calculating a correction factor based on the reward at time t+1 minus and the risk aversion coefficient until the end state is met.

### Claim 2 (depends on 1)

The method of claim 1, wherein the Q table is initialized to an arbitrary value.

### Claim 3 (depends on 2)

The method of claim 2, wherein the Q table maps a plurality of actions and a plurality of results.

### Claim 4 (depends on 1)

The method of claim 1, wherein the end state is based on a number of time steps.

### Claim 5 (depends on 1)

The method of claim 1, wherein the end state is based on a condition being reached.

### Claim 6 (depends on 1)

The method of claim 1, wherein the condition is based on the correction factor changing by less than a predetermined amount.

### Claim 7 (depends on 1)

The method of claim 1, wherein the correction factor is calculated using the following equation:
α
t
(
R
t
+
1
-
β
2
⁢
(
R
t
+
1
-
R
_
⁡
(
s
t
,
α
t
)
)
2
where:
αt is a learning rate;
R is the reward;
β is the risk aversion coefficient; and
R is the average reward over time t.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
outputting a risk-sensitive policy including the correction factor;
receiving real-time state data and a plurality of actions;
selecting an action from the plurality of actions using the risk-sensitive policy including the correction factor

### Claim 9 (depends on 1)

The method of claim 1, wherein the action comprises at least one of a pricing and a trade.

### Claim 10 (depends on 1)

The method of claim 1, wherein the reward comprises a profit or a loss.

### Claim 11

A method for a method for training a risk-sensitive reinforcement learning policy, comprising:
a risk-sensitive learning engine comprising at least one computer processor calculating, from an initial distribution parameter, an experience replay table, a number of distribution gradient steps, and a number of Stochastic Gradient Descent (SGD) samples, calculating an approximation of an optimal distribution parameter;
the risk-sensitive learning engine receiving, from a data source, a plurality of sets of training data for a plurality of time steps, each set of training data comprising an initial state, an action, a reward, and a state at the next time step;
the risk-sensitive learning engine receiving a risk aversion coefficient; and
for each time step t:
the risk-sensitive learning engine calculating, for a training budget comprising a number of episodes b, a parametric approximation of an average reward over time for the training data;
the risk-sensitive learning engine calculating a correction factor based on the reward at time t+1, the parametric approximation of the average reward over time, and the risk aversion coefficient, wherein the correction factor minimizes stochasticity for the reward based on the risk aversion coefficient; and
increasing time t to t+1, executing an action for time+1 and monitoring the reward at time t+1 and a state at time t+1, calculating the parametric approximation of an average reward over time t, and calculating the correction factor based on the reward at time t+1 minus and the risk aversion coefficient until an end state is met.

### Claim 12 (depends on 11)

The method of claim 11, wherein the end state is based on a number of time steps.

### Claim 13 (depends on 11)

The method of claim 11, wherein the end state is based on a condition being reached.

### Claim 14 (depends on 11)

The method of claim 11, wherein the condition is based on the correction factor changing by less than a predetermined amount.

### Claim 15 (depends on 11)

The method of claim 11, wherein the correction factor is calculated using the equations:
υ
t
,
b
:=
∑
t
′
=
t
T
b
-
1
⁢
γ
t
′
-
t
⁢
R
t
′
+
1
(
b
)
-
β
2
⁢
γ
2
⁢
(
t
′
-
t
)
⁡
(
R
t
′
+
1
(
b
)
-
R
^
ψ
⁡
(
s
t
′
(
b
)
,
α
t
′
(
b
)
)
)
2
V
b
←
∑
t
=
0
T
b
-
1
⁢
∇
ln
⁢
⁢
π
θ
n
⁡
(
α
t
(
b
)
❘
s
t
(
b
)
)
⁢
υ
t
,
b
where:
β is the risk aversion coefficient;
R is the reward;
t′ is a time index based on time t
{circumflex over (R)} is the parametric approximation of the average reward over time t;
s is a state at time index t′;
a is an action at time index t′; and
b is the number of episodes.

### Claim 16 (depends on 11)

The method of claim 11, further comprising:
outputting a risk-sensitive policy including the correction factor;
receiving real-time state data and a plurality of actions;
selecting an action from the plurality of actions using the risk-sensitive policy including the correction factor

### Claim 17 (depends on 11)

The method of claim 11, wherein the action comprises at least one of a pricing and a trade.

### Claim 18 (depends on 11)

The method of claim 11, wherein the reward comprises a profit or a loss.
