Method of generating training data for training a neural network, method of training a neural network and using neural network for autonomous operations

A method of training a neural network, comprising:

• (i) generating a first set of policy values Q(si,ai) for each state-action pair si, ai in a sample data set D {(si, ai, si+1,ri)} using an action-value function denoted a Q function, wherein si is a current state of an object in an environment, ai is an action chosen for the current state, si+1 is a subsequent state of the object in the environment and ri is a reward value for taking the action, ai, in a current state, si, a value of which is determined in accordance with a reward function;
• (ii) generating a second set of policy values Q (si+1,a) for each subsequent state si+1 for all tuples (si, ai, si+1,ri) in the sample data set D for each action in a set of all possible actions using the Q function;
• (iii) generating an approximate action-value function, denoted a Q* function and that approximates policy values of a state-action pair (s, a) following a policy value function, from the first set of policy values Q(si,ai) for the current state si and the action ai selected for the current state si and the second set of policy values Q (si+1,a) for the subsequent state si+after the action ai, wherein generating the Q* function using an approximate policy iteration (API) procedure by:
• initializing a matrix A and a vector b;
• for each tuple (si, aisi+1,ri) in the sample data set D:
• selecting an action, a*, that results in maximum value of Q(si+i,a) from a set of all possible actions (a*=argmaxaQ (si+1, a));
• generating a vector from an output layer of the neural network φ(si, ai), φ*(si+1,a*) using ϕ(s) and tabular action;
• and
• updating a value of the matrix A and the vector b using the following equations
• A=A+φ(si,ai)(γφ(si+1,a*)−φ(si,ai))T,
• b=b+φ(si,ai)ri,
• wherein γ is a discount factor between 0 and 1;
• and
• calculating a weight vector w according to the following equation:
• ω=−A−1b;
• (iv) generating a training target for the neural network using the Q* function;
• (v) calculating a training error as a difference between the training target and the policy value Q (si, ai) for a corresponding state-action pair in the sample data set D;
• and
• (vi) training the neural network by updating at least some parameters of the neural network to minimize the training error through back propagation.

A non-transitory machine readable medium having tangibly stored thereon executable instructions for execution by a processor of a computing device, wherein the executable instructions, when executed by the processor of the computing device, cause the computing device to:

• (i) generate a first set of policy values Q(si,ai) for each state-action pair si, ai in a sample data set D {(si, ai, si+1,ri)} using an action-value function denoted a Q function, wherein si is a current state of an object in an environment, ai is an action chosen for the current state, si+1 a subsequent state of the object in the environment and ri is a reward value for taking the action, ai, in a currrent state, si, a value of which is determined in accordance with a reward function;
• (ii) generate a second set of policy values Q (si+1,a) for each subsequent state si+1 for all tuples (si, ai, si+1,ri) in the sample data set D for each action in a set of all possible actions using the Q function;
• (iii) generate an approximate action-value function, denoted a Q* function, and that approximates policy values of a state-action pair (s, a) following a policy value function, from the first set of policy values Q(si,ai) for the current state si and the action ai selected for the current state si and the second set of policy values Q (si+1,a) for the subsequent state si+1 after the action ai, the neural network configured to generate the Q* function using an approximate policy iteration (API) procedure by:
• initializing a matrix A and a vector b;
• for each tuple (si, ai, si+1,ri) in the sample data set D:
• selecting an action, a*, that results in maximum value of Q (si+i, a) from a set of all possible actions (a*=argmaxaQ (si+1, a));
• generating a vector from an output layer of the neural network φ(si,ai), φ*(si+1,a*) using ϕ(s) and tabular action;
• updating a value of the matrix A and the vector b using the following equations
• A=A+φ(si,ai)(γφ(si+1,a*)−φ(si,ai))T,
• b=b+φ(si,ai)ri,
• wherein y is a discount factor between 0 and 1;
• and
• calculating a weight vector co according to the following equation:
• ω=−A−1b;
• (iv) generate a training target for a neural network using the Q*-function;
• (v) calculate a training error as a difference between the training target and a policy value Q (si, ai)-for a corresponding state-action pair in the sample data set D;
• and
• (vi) train the neural network by updating at least some parameters of the neural network to minimize the training error through back propagation.