Intelligent control with hierarchical stacked neural networks

A neural network method, comprising:

• modeling an environment of an agent as a finite state Markov decision process having a set of environment states S dependent on actions A of the agent, with at least one automated processor;
• implementing a policy πt for operation of the agent within the environment based on the set of environment states S and an estimate of at least one dynamic parameter of the environment, to perform at time t an action At of a set of possible actions A(st) by the agent, the policy πt being defined as a conditional distribution over the set of possible actions A(st) given an observation of state st∈S, by the agent operating within the environment, with probability distributions:
• instantaneous cost distribution P(rt|st),
• observation distribution P(A(st)|st), and
• transition distribution P(st+1|st,A(st))
• receiving the observation of the state st+1 and a temporally-associated cost or reward rt+1 resulting from operation of the agent in the environment controlled according to the policy πt;
• updating the policy πt with the at least one automated processor to produce updated policy πt+1, dependent on the received observation st+1 and the temporally-associated cost or reward rt+1, to improve the policy πt to optimize an expected future cumulative cost or reward
• R
• =
• ∑
• t
• =
• 0
• ∞
• γ
• t
• ⁢
• r
• t
• ,
• where 0≤γ≤1 is a future cost or reward discounting factor,
• the policy πt representing a set of parameters defining an artificial neural network having a plurality of hierarchical layers, and having at least one layer of neurons which:
• receives inputs representing the received observation of the state st processed by neurons of another hierarchical layer, and
• produces outputs to neurons of another hierarchical layer which generates the action, and
• the plurality of hierarchical layers being trained according to respectfully distinct training criteria;
• and
• performing, with the at least one automated processor, a classification analysis to produce a noise vector representing information within a respective set of inputs of at least one of the hierarchical layers which fails to specifically excite or suppress a respective output of the at least one of the hierarchical layers of the artificial neural network.

A non-transitory computer readable medium storing therein instructions for causing a programmable automated processor to perform a method comprising:

• instructions for modeling an environment altered by actions of an agent as a finite state Markov decision process having a set of environment states, comprising estimating at least one dynamic parameter of the set of environment states;
• instructions for implementing a policy for operation of the agent within the environment based on the set of environment states, to perform an action by the agent within the environment, the policy being defined as a conditional distribution over actions given an observation of state, by the agent operating within the environment,
• the policy representing a set of parameters stored in a memory defining an artificial neural network having a plurality of hierarchical layers and having at least one layer of neurons which receives inputs representing the received observation of the state indirectly from other neurons, and produces outputs to neurons of another hierarchical layer which implement the action, the plurality of hierarchical layers being trained according to respectfully distinct training criteria;
• instructions for receiving an observation of the state and a temporally-associated cost or reward resulting from operation of the agent in the environment controlled according to the policy;
• instructions for updating the policy, dependent on the received observation and the temporally-associated cost or reward, to improve the policy to optimize an expected future cumulative cost or reward;
• and
• instructions for performing a classification analysis to produce a noise vector representing information within a respective set of inputs of at least one of the hierarchical layers which fails to specifically excite or suppress a respective output of the at least one of the hierarchical layers of the artificial neural network.