One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
- during execution of an autonomous agent configured to control operation of a physical mechanism, obtaining one or more current observations of a physical environment;
- based at least on one of the one or more current observations of the physical environment, generating a trajectory tree that represents a plurality of possible trajectories in a belief space, wherein a belief about a discrete latent state can be represented by a probability distribution over the discrete latent state,
- wherein a plurality of nodes of the trajectory tree represents values of one or more continuous controls used by the autonomous agent to control the physical mechanism,
- wherein branches between nodes represent possible outcomes, possible observations, and beliefs over the one or more discrete latent states;
- wherein a first node of the plurality of nodes represents a first control, and a second node of the plurality of nodes represents a second control which is a function of at least the values of a first possible outcome of the possible outcomes, a first possible observation of the possible observations, and a first belief, of the beliefs over the one or more discrete latent states,
- wherein the first belief, the first possible outcome, and the first possible observation correspond to a first branch between the first and second node, and
- wherein each node of the plurality of nodes is associated with one of a plurality of timesteps along the plurality of possible trajectories;
- wherein generating the trajectory tree comprises:
- (a) in a first forward pass, initializing one or more values of the one or more continuous controls for the plurality of nodes;
- (b) responsive to completing the first forward pass, in a backward pass, starting at a plurality of leaf nodes in the plurality of nodes, propagating results of a derivative function from the plurality of leaf nodes, through one or more parent nodes, to a root node, where a parent node of a leaf node is a node in the plurality of nodes sharing a branch with the leaf node, and a root node is a node in the plurality of nodes without a parent node,
- wherein the derivative function comprises computing a derivative of a value function with respect to a state or control,
- wherein the value function comprises computing an expected sum of costs of each node in a trajectory up to the finite horizon of the trajectory tree, the expected sum of costs corresponding to the value of the trajectory,
- wherein a cost of a node in a trajectory is a value indicating the undesirability of a control value at the node of the trajectory;
- (c) responsive to completing the backward pass, in a second forward pass, applying an optimization function to values of the one or more continuous controls, based at least on the results of the derivative function;
- and
- (d) updating the values of the one or more continuous controls by applying the locally optimal control modification to the values of the one or more continuous controls;
- and
- determining a current value of a continuous control of the one or more continuous controls associated with a current node of the plurality of nodes;
- and
- applying the current value of the continuous control to operation of the physical mechanism.