# US 11,308,401 B2 · Interactive reinforcement learning with dynamic reuse of prior knowledge

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,401 B2
- Document ID: US11308401B2
- Status: Granted
- Kind: B2
- Application number: 16/263,930
- Filed: 2019-01-31
- Published: 2019-08-01
- Granted: 2022-04-19
- Earliest claimed: 2018-01-31
- Assignees: Royal Bank of Canada
- Applicants: Royal Bank of Canada
- Inventors: Matthew Edmund Taylor; Zhaodong Wang
- Web page: https://patentgrep.com/patent/US11308401B2

## CPC

- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G06F 17/17: Function evaluation by approximation methods, e.g. inter- or extrapolation, smoothing, least mean square method ({G06F17/18 takes precedence } ; interpolation for numerical control G05B19/18)
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 3/096: Transfer learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/00: Machine learning
- G06N 3/045: Combinations of networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems, methods, and computer readable media directed to interactive reinforcement learning with dynamic reuse of prior knowledge are described in various embodiments. The interactive reinforcement learning is adapted for providing computer implemented systems for dynamic action selection based on confidence levels associated with demonstrator data or portions thereof.

## Claims

### Claim 1

A system for biasing a machine learning architecture using one or more demonstrator data sets, the machine learning architecture for controlling one or more actions conducted by an agent in an environment which transitions between one or more states, the system comprising:
a physical computer processor operating in conjunction with computer memory and computer storage, the processor configured to provide:
a receiver configured to obtain one or more demonstrator data sets, each demonstrator data set including a data structure representing one or more state-action pairs observed in one or more interactions with the environment;
a data storage configured to maintain, for each demonstrator data set or sub-portions thereof, one or more confidence data values, associated with at least one state of the one or more states;
a supervised classifier for training using the one or more demonstrator data sets or sub-portions thereof;
an action execution processor configured to generate control signals for executing an action associated with an action-source selected from at least one of the one or more demonstrator data sets based on the supervised classifier or an internal policy function maintained by the machine learning architecture, the selecting based at least upon the one or more confidence data values; and
a state observer configured to monitor a new state resulting from the execution of the action and an associated reward outcome; and to update the internal policy function maintained by the machine learning architecture based at least on the observed reward outcome;
wherein the one or more confidence data values are generated using a dynamic temporal difference confidence measurement; and wherein the dynamic temporal difference confidence measurement is based on the relation:
C(s)←(1−F(α))XC(s)+F(α)X[F(r)+γXC(s′)]
where γ is a discount factor, r is a reward function, and α is an update parameter.

### Claim 2 (depends on 1)

The system of claim 1, wherein the state observer is configured to update at least one of the confidence data values of the one or more confidence data values based on the observed reward outcome.

### Claim 3 (depends on 1)

The system of claim 1, wherein the temporal difference confidence measurement includes a dynamic rate update function based on the relation:
F
⁡
(
α
)
=
α
×
max
⁢
{
1
Σ
i
⁢
⁢
exp
⁡
(
θ
i
T
·
x
)
⁡
[
exp
⁡
(
θ
1
T
·
x
)
)
exp
⁡
(
θ
2
T
·
x
)
)
…
exp
⁡
(
θ
i
T
·
x
)
)
]
}
.

### Claim 4 (depends on 1)

The system of claim 1, wherein the temporal difference confidence measurement includes a dynamic confidence update function based on the relation:
F
⁡
(
r
)
=
r
r_max
×
max
⁢
{
1
Σ
i
⁢
⁢
exp
⁡
(
θ
i
T
·
x
)
⁡
[
exp
⁡
(
θ
1
T
·
x
)
)
exp
⁡
(
θ
2
T
·
x
)
)
…
exp
⁡
(
θ
i
T
·
x
)
)
]
}
wherein
r
r_max
is a normalized reward.

### Claim 5 (depends on 1)

The system of claim 1, wherein the selecting of the action-source is based upon an action selection mechanism including a hard decision architecture adapted for maximizing a current confidence expectation.

### Claim 6 (depends on 1)

The system of claim 1, wherein the selecting of the action-source is based upon an action selection mechanism including a soft decision architecture adapted for maximizing a current confidence expectation, the soft decision architecture normalizing a confidence in prior knowledge score (CP(s)) and a confidence in machine learning architecture policy function (CQ(s)) based on the relation:
tanh
⁡
(
rCQ
)
=
e
rCQ
-
e
-
rCQ
e
rCQ
+
e
-
rCQ
;
and the action-source is determined based the relation:
AS
=
{
Q
⁢
P
=
tanh
⁡
(
rCQ
)
+
1
tanh
⁡
(
rCP
)
+
tanh
⁡
(
rCQ
)
+
2
Prior
P
=
tanh
⁡
(
rCP
)
+
1
tanh
⁡
(
rCP
)
+
tanh
⁡
(
rCQ
)
+
2
.
(
1
)

### Claim 7 (depends on 1)

The system of claim 1, wherein the selecting of the action-source is based upon an action selection mechanism including a soft-hard-ε decision architecture including an ε-greedy switch for greedily exploiting a determined confidence value while performing probabilistic exploration.

### Claim 8 (depends on 1)

The system of claim 1, wherein the one or more demonstrator data sets are a plurality of demonstrator data sets, including at least a first demonstrator data set provided from a first demonstrator and a second demonstrator data set provided from a second demonstrator, and the selection of the action-source includes selecting at least from between the first demonstrator data set and the second demonstrator data set.

### Claim 9 (depends on 1)

The system of claim 1, wherein upon the confidence data value associated with a specific demonstrator data set or a portion thereof is detected to be reduced beyond a threshold value, a notification engine is configured to generate a notification requesting improved demonstration data sets from an associated source of demonstration data sets.

### Claim 10

A method of biasing a machine learning architecture using one or more demonstrator data sets, and the machine learning architecture for controlling one or more actions conducted by an agent in an environment which transitions between one or more states, the method comprising:
receiving the one or more demonstrator data sets, each demonstrator data set including a data structure representing one or more state-action pairs observed in one or more interactions with the environment;
for each demonstrator data set or sub-portions thereof, maintaining one or more confidence data values, associated with at least one state of the one or more states;
training a supervised classifier using the one or more demonstrator data sets or sub-portions thereof;
executing an action associated with an action-source selected from at least one of the one or more demonstrator data sets based on the supervised classifier or an internal policy function maintained by the machine learning architecture, the selecting based at least upon the one or more confidence data values;
observing a new state resulting from the execution of the action and an associated reward outcome; and
updating the internal policy function maintained by the machine learning architecture based at least on the observed reward outcome;
wherein the one or more confidence data values are generated using a dynamic temporal difference confidence measurement and; wherein the dynamic temporal difference confidence measurement is based on the relation:
C(s)←(1−F(α))XC(s)+F(α)X[F(r)+γXC(s′)]
where γ is a discount factor, r is a reward function, and α is an update parameter.

### Claim 11 (depends on 10)

The method of claim 10, comprising:
updating at least one of the confidence data values of the one or more confidence data values based on the observed reward outcome.

### Claim 12 (depends on 10)

The method of claim 10, wherein the temporal difference confidence measurement includes a dynamic rate update function based on the relation:
F
⁡
(
α
)
=
α
×
max
⁢
{
1
Σ
i
⁢
⁢
exp
⁡
(
θ
i
T
·
x
)
⁡
[
exp
⁡
(
θ
1
T
·
x
)
)
exp
⁡
(
θ
2
T
·
x
)
)
…
exp
⁡
(
θ
i
T
·
x
)
)
]
}
.

### Claim 13 (depends on 10)

The method of claim 10, wherein the temporal difference confidence measurement includes a dynamic confidence update function based on the relation:
F
⁡
(
r
)
=
r
r_max
×
max
⁢
{
1
Σ
i
⁢
⁢
exp
⁡
(
θ
i
T
·
x
)
⁡
[
exp
⁡
(
θ
1
T
·
x
)
)
exp
⁡
(
θ
2
T
·
x
)
)
…
exp
⁡
(
θ
i
T
·
x
)
)
]
}
wherein
r
r_max
is a normalized reward.

### Claim 14 (depends on 10)

The method of claim 10, wherein the selecting of the action-source is based upon an action selection mechanism including a hard decision architecture adapted for maximizing a current confidence expectation.

### Claim 15 (depends on 10)

The method of claim 10, wherein the selecting of the action-source is based upon an action selection mechanism including a soft decision architecture adapted for maximizing a current confidence expectation, the soft decision architecture normalizing a confidence in prior knowledge score (CP(s)) and a confidence in machine learning architecture policy function (CQ(s)) based on the relation:
tanh
⁡
(
rCQ
)
=
e
rCQ
-
e
-
rCQ
e
rCQ
+
e
-
rCQ
;
and the action-source is determined based the relation:
AS
=
{
Q
⁢
P
=
tanh
⁡
(
rCQ
)
+
1
tanh
⁡
(
rCP
)
+
tanh
⁡
(
rCQ
)
+
2
Prior
P
=
tanh
⁡
(
rCP
)
+
1
tanh
⁡
(
rCP
)
+
tanh
⁡
(
rCQ
)
+
2
.
(
1
)

### Claim 16 (depends on 10)

The method of claim 10, wherein the selecting of the action-source is based upon an action selection mechanism including a soft-hard-ε decision architecture including an ε-greedy switch for greedily exploiting a determined confidence value while performing probabilistic exploration.

### Claim 17 (depends on 10)

The method of claim 10, wherein if the confidence data value associated with a specific demonstrator data set or a portion thereof is reduced beyond a threshold value, the method comprises generating a notification requesting improved demonstration data sets from an associated source of demonstration data sets.

### Claim 18

A non-transitory computer readable medium storing machine interpretable instructions, which when executed, cause a processor to perform a method of biasing a machine learning architecture using one or more demonstrator data sets, and the machine learning architecture for controlling one or more actions conducted by an agent in an environment which transitions between one or more states, the method comprising:
receiving the one or more demonstrator data sets, each demonstrator data set including a data structure representing one or more state-action pairs observed in one or more interactions with the environment;
for each demonstrator data set or sub-portions thereof, maintaining one or more confidence data values, associated with at least one state of the one or more states;
training a supervised classifier using the one or more demonstrator data sets or sub-portions thereof;
executing an action associated with an action-source selected from at least one of the one or more demonstrator data sets based on the supervised classifier or an internal policy function maintained by the machine learning architecture, the selecting based at least upon the one or more confidence data values;
observing a new state resulting from the execution of the action and an associated reward outcome; and
updating the internal policy function maintained by the machine learning architecture based at least on the observed reward outcome;
wherein the one or more confidence data values are generated using a dynamic temporal difference confidence measurement; and wherein the dynamic temporal difference confidence measurement is based on the relation:
C(s)←(1−F(α))XC(s)+F(α)X[F(r)+γXC(s′)]
where γ is a discount factor, r is a reward function, and α is an update parameter.