Method and system for improving a policy for a stochastic control problem

A digital computer comprising:

• a central processing unit;
• a display device;
• a communication port for operatively connecting the digital computer to a sampling device coupled to a digital computer and to a sampling device control system;
• a memory unit comprising an application for improving a policy for a stochastic control problem, the stochastic control problem being characterized by a set of actions, a set of states, a reward structure as a function of states and actions, and a plurality of decision epochs, wherein evolution of an underlying stochastic state process depends on a plurality of actions in a policy, the application comprising:
• instructions for using a sampling device coupled to the digital computer and to a sampling device control system, the sampling device obtaining data representative of sample configurations of a Boltzmann machine comprising a plurality of nodes, a plurality of couplers, a plurality of biases, each bias corresponding to a node in the plurality of nodes, a plurality of coupling weights, each coupling weight corresponding to a coupler of the plurality of couplers, and a transverse field strength;
• instructions for obtaining, using the digital computer, initialization data comprising the set of actions, the set of states, the reward structure of the stochastic control problem and an initial policy for the stochastic control problem, the policy comprising a choice of at least one action for each state;
• instructions for using the digital computer and the sampling device control system, assigning data representative of an initial weight and a bias of respectively each coupler and each node and the transverse field strength of the Boltzmann machine to the sampling device;
• instructions for, until a stopping criterion is met:
• generating a present-epoch state-action pair using the digital computer,
• using the digital computer and the sampling device control system, amending data representative of none or at least one coupler and at least one bias using the generated present-epoch state-action pair,
• performing a sampling corresponding to the present-epoch state-action pair to obtain first sampling empirical means,
• obtaining, using the first sampling empirical means, using the digital computer, an approximation of a value of a Q-function at the present-epoch state-action, the value of the Q-function being representative of a utility of the present-epoch state-action pair,
• obtaining, using the digital computer, a future-epoch state-action pair, wherein the state is obtained through a stochastic state process, and further wherein the obtaining of the action comprises performing a stochastic optimization test on the plurality of all state-action pairs comprising the future-epoch state and any possible action to thereby provide the action at the future-epoch and update the policy for the future-epoch state,
• amending, using the digital computer and the sampling device control system, data representative of none or at least one coupler and at least one bias using the generated future-epoch state-action pair,
• performing a sampling corresponding to the future-epoch state-action pair to obtain second sampling empirical means,
• obtaining, using the second sampling empirical means, using the digital computer, an approximation of a value of the Q-function at the future-epoch state-action, the value of the Q-function being representative of a utility of the future-epoch state-action pair, and
• using the digital computer, updating each weight and each bias of respectively each coupler and each node of the Boltzmann machine using the generated approximations of the value of the Q-function and the first sampling empirical means at present-epoch state-action pair and a corresponding reward at the present-epoch state-action pair obtained using the reward structure;
• and
• instructions for providing the policy using the digital computer when the stopping criterion is met.

A non-transitory computer readable storage medium for storing computer-executable instructions which, when executed, cause a digital computer to perform a method for improving a policy for a stochastic control problem, the stochastic control problem being characterized by a set of actions, a set of states, a reward structure as a function of states and actions, and a plurality of decision epochs, wherein evolution of an underlying stochastic state process depends on a plurality of actions in a policy, the method comprising:

• using a sampling device coupled to a digital computer and to a sampling device control system, the sampling device obtaining data representative of sample configurations of a Boltzmann machine comprising:
• a plurality of nodes,
• a plurality of couplers,
• a plurality of biases, each bias corresponding to a node in the plurality of nodes,
• a plurality of coupling weights, each coupling weight corresponding to a coupler of the plurality of couplers, and
• a transverse field strength;
• obtaining, using the digital computer, initialization data comprising the set of actions, the set of states, the reward structure of the stochastic control problem and an initial policy for the stochastic control problem, the policy comprising a choice of at least one action for each state;
• using the digital computer and the sampling device control system, assigning data representative of an initial weight and a bias of respectively each coupler and each node and the transverse field strength of the Boltzmann machine to the sampling device;
• until a stopping criterion is met:
• generating a present-epoch state-action pair using the digital computer,
• using the digital computer and the sampling device control system, amending data representative of none or at least one coupler and at least one bias using the generated present-epoch state-action pair,
• performing a sampling corresponding to the present-epoch state-action pair to obtain first sampling empirical means,
• obtaining, using the first sampling empirical means, using the digital computer, an approximation of a value of a Q-function at the present-epoch state-action, the value of the Q-function being representative of a utility of the present-epoch state-action pair,
• obtaining, using the digital computer, a future-epoch state-action pair, wherein the state is obtained through a stochastic state process, and further wherein the obtaining of the action comprises performing a stochastic optimization test on the plurality of all state-action pairs comprising the future-epoch state and any possible action to thereby provide the action at the future-epoch and update the policy for the future-epoch state,
• amending, using the digital computer and the sampling device control system, data representative of none or at least one coupler and at least one bias using the generated future-epoch state-action pair,
• performing a sampling corresponding to the future-epoch state-action pair to obtain second sampling empirical means,
• obtaining, using the second sampling empirical means, using the digital computer, an approximation of a value of the Q-function at the future-epoch state-action, the value of the Q-function being representative of a utility of the future-epoch state-action pair,
• using the digital computer, updating each weight and each bias of respectively each coupler and each node of the Boltzmann machine using the generated approximations of the value of the Q-function and the first sampling empirical means at present-epoch state-action pair and a corresponding reward at the present-epoch state-action pair obtained using the reward structure;
• and
• providing the policy using the digital computer when the stopping criterion is met.