Interactive reinforcement learning with dynamic reuse of prior knowledge
AssigneeRoyal Bank of Canada
InventorsMatthew Edmund Taylor, Zhaodong Wang
Life of this patent
Estimating the term…
Abstract
Systems, methods, and computer readable media directed to interactive reinforcement learning with dynamic reuse of prior knowledge are described in various embodiments. The interactive reinforcement learning is adapted for providing computer implemented systems for dynamic action selection based on confidence levels associated with demonstrator data or portions thereof.
Claims18
A system for biasing a machine learning architecture using one or more demonstrator data sets, the machine learning architecture for controlling one or more actions conducted by an agent in an environment which transitions between one or more states, the system comprising:
- a physical computer processor operating in conjunction with computer memory and computer storage, the processor configured to provide:
- a receiver configured to obtain one or more demonstrator data sets, each demonstrator data set including a data structure representing one or more state-action pairs observed in one or more interactions with the environment;
- a data storage configured to maintain, for each demonstrator data set or sub-portions thereof, one or more confidence data values, associated with at least one state of the one or more states;
- a supervised classifier for training using the one or more demonstrator data sets or sub-portions thereof;
- an action execution processor configured to generate control signals for executing an action associated with an action-source selected from at least one of the one or more demonstrator data sets based on the supervised classifier or an internal policy function maintained by the machine learning architecture, the selecting based at least upon the one or more confidence data values;
- and
- a state observer configured to monitor a new state resulting from the execution of the action and an associated reward outcome;
- and to update the internal policy function maintained by the machine learning architecture based at least on the observed reward outcome;
- wherein the one or more confidence data values are generated using a dynamic temporal difference confidence measurement;
- and wherein the dynamic temporal difference confidence measurement is based on the relation:
- C(s)←(1−F(α))XC(s)+F(α)X[F(r)+γXC(s′)]
- where γ is a discount factor, r is a reward function, and α is an update parameter.
A method of biasing a machine learning architecture using one or more demonstrator data sets, and the machine learning architecture for controlling one or more actions conducted by an agent in an environment which transitions between one or more states, the method comprising:
- receiving the one or more demonstrator data sets, each demonstrator data set including a data structure representing one or more state-action pairs observed in one or more interactions with the environment;
- for each demonstrator data set or sub-portions thereof, maintaining one or more confidence data values, associated with at least one state of the one or more states;
- training a supervised classifier using the one or more demonstrator data sets or sub-portions thereof;
- executing an action associated with an action-source selected from at least one of the one or more demonstrator data sets based on the supervised classifier or an internal policy function maintained by the machine learning architecture, the selecting based at least upon the one or more confidence data values;
- observing a new state resulting from the execution of the action and an associated reward outcome;
- and
- updating the internal policy function maintained by the machine learning architecture based at least on the observed reward outcome;
- wherein the one or more confidence data values are generated using a dynamic temporal difference confidence measurement and;
- wherein the dynamic temporal difference confidence measurement is based on the relation:
- C(s)←(1−F(α))XC(s)+F(α)X[F(r)+γXC(s′)]
- where γ is a discount factor, r is a reward function, and α is an update parameter.
A non-transitory computer readable medium storing machine interpretable instructions, which when executed, cause a processor to perform a method of biasing a machine learning architecture using one or more demonstrator data sets, and the machine learning architecture for controlling one or more actions conducted by an agent in an environment which transitions between one or more states, the method comprising:
- receiving the one or more demonstrator data sets, each demonstrator data set including a data structure representing one or more state-action pairs observed in one or more interactions with the environment;
- for each demonstrator data set or sub-portions thereof, maintaining one or more confidence data values, associated with at least one state of the one or more states;
- training a supervised classifier using the one or more demonstrator data sets or sub-portions thereof;
- executing an action associated with an action-source selected from at least one of the one or more demonstrator data sets based on the supervised classifier or an internal policy function maintained by the machine learning architecture, the selecting based at least upon the one or more confidence data values;
- observing a new state resulting from the execution of the action and an associated reward outcome;
- and
- updating the internal policy function maintained by the machine learning architecture based at least on the observed reward outcome;
- wherein the one or more confidence data values are generated using a dynamic temporal difference confidence measurement;
- and wherein the dynamic temporal difference confidence measurement is based on the relation:
- C(s)←(1−F(α))XC(s)+F(α)X[F(r)+γXC(s′)]
- where γ is a discount factor, r is a reward function, and α is an update parameter.
Dependencies are inferred from the claim text.