# US 11,298,821 B1 · Systems and methods for learning and generating movement policies for a dynamical system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,298,821 B1 - Document ID: US11298821B1 - Status: Granted - Kind: B1 - Application number: 16/837,548 - Filed: 2020-04-01 - Granted: 2022-04-12 - Earliest claimed: 2018-10-16 - Assignees: University of South Florida - Applicants: University of South Florida - Inventors: Yu Sun; Yongqiang Huang - Web page: https://patentgrep.com/patent/US11298821B1 ## CPC - B25J 9/163: {learning, adaptive, model based, rule based expert control} - B25J 9/1661: {characterised by task planning, object-oriented languages} - B25J 9/1664: {characterised by motion, path, trajectory planning} - G05B 13/027: {using neural networks only} - G05B 19/42: Recording and playback systems, i.e. in which the program is recorded from a cycle of operations, e.g. the cycle of operations being manually controlled, after which this record is played back on the same machine - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 3/092: Reinforcement learning - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06N 20/00: Machine learning - G06N 3/047: Probabilistic or stochastic networks - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract System and methods for generating a trajectory of a dynamical system are described herein. An example method includes modelling a policy from demonstration data. The method also includes generating a first set of action particles by sampling from the policy, where each of the action particles in the first set includes a respective system action, predicting a respective outcome of the dynamical system in response to each of the action particles in the first set, and weighting each of the action particles in the first set according to a respective probability of achieving a desired outcome. The method further includes generating a second set of action particles by sampling from the weighted action particles, where each of the action particles in the second set includes a respective system action, and selecting a next system action in the trajectory of the dynamical system from the action particles in the second set. ## Claims ### Claim 1 A computer-implemented method for generating a trajectory of a dynamical system, comprising: modelling a policy from demonstration data, wherein the policy is a distribution of a plurality of system actions conditioned on a plurality of system states and outcomes; generating a first set of action particles by sampling from the policy, wherein each of the action particles in the first set includes a respective system action; predicting a respective outcome of the dynamical system in response to each of the action particles in the first set; weighting each of the action particles in the first set according to a respective probability of achieving a desired outcome of the dynamical system, wherein the respective probabilities of achieving the desired outcome are determined using the respective outcome of the dynamical system in response to each of the action particles in the first set; generating a second set of action particles by sampling from the weighted action particles, wherein each of the action particles in the second set includes a respective system action; and selecting a next system action in the trajectory of the dynamical system from the action particles in the second set. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the policy is modeled using a recurrent mixture density network (RMDN). ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein weighting each of the action particles in the first set according to a respective probability of achieving a desired outcome of the dynamical system comprises: calculating a respective reward value for each of the action particles in the first set by comparing a respective outcome of the dynamical system in response to each of the action particles in the first set with the desired outcome of the dynamical system; and weighting each of the action particles in the first set by a respective reward value. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein selecting a next system action in the trajectory of the dynamical system from the action particles in the second set comprises: estimating a probability density function of the action particles in the second set; and selecting the next system action using the probability density function. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein generating a second set of action particles by sampling from the weighted action particles comprises importance sampling from the weighted action particles. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the respective outcome of the dynamical system in response to each of the action particles in the first set is associated with a confidence value. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein the respective outcome of the dynamical system in response to each of the action particles in the first set is predicted using a recurrent mixture density network (RMDN). ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the desired outcome is defined by a user. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the policy is represented in a stochastic fashion. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the trajectory comprises a plurality of sequential system states over time. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein the dynamical system is a robotic system. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein the demonstration data is human demonstration data. ### Claim 13 A system for generating a trajectory of a dynamical system, comprising: a recurrent mixture density network (RMDN) configured to model a policy from demonstration data, wherein the policy is a distribution of a plurality of system actions conditioned on a plurality of system states and outcomes; and a processor and a memory operably connected to the processor, wherein the memory has computer-executable instructions stored thereon that, when executed by the processor, cause the processor to: generate a first set of action particles by sampling from the policy, wherein each of the action particles in the first set includes a respective system action; predict a respective outcome of the dynamical system in response to each of the action particles in the first set; weight each of the action particles in the first set according to a respective probability of achieving a desired outcome of the dynamical system, wherein the respective probabilities of achieving the desired outcome are determined using the respective outcome of the dynamical system in response to each of the action particles in the first set; generate a second set of action particles by sampling from the weighted action particles, wherein each of the action particles in the second set includes a respective system action; and select a next system action in the trajectory of the dynamical system from the action particles in the second set. ### Claim 14 (depends on 13) The system of claim 13, further comprising a robot, wherein the memory has further computer-executable instructions stored thereon that, when executed by the processor, cause the processor to transmit a command including the next system action to the robot. ### Claim 15 (depends on 13) The system of claim 13, wherein the policy is modeled using a recurrent mixture density network (RMDN). ### Claim 16 (depends on 13) The system of claim 13, wherein weighting each of the action particles in the first set according to a respective probability of achieving a desired outcome of the dynamical system comprises: calculating a respective reward value for each of the action particles in the first set by comparing a respective outcome of the dynamical system in response to each of the action particles in the first set with the desired outcome of the dynamical system; and weighting each of the action particles in the first set by a respective reward value. ### Claim 17 (depends on 13) The system of claim 13, wherein selecting a next system action in the trajectory of the dynamical system from the action particles in the second set comprises: estimating a probability density function of the action particles in the second set; and selecting the next system action using the probability density function. ### Claim 18 (depends on 13) The system of claim 13, wherein generating a second set of action particles by sampling from the weighted action particles comprises importance sampling from the weighted action particles. ### Claim 19 (depends on 13) The system of claim 13, wherein the respective outcome of the dynamical system in response to each of the action particles in the first set is associated with a confidence value. ### Claim 20 (depends on 13) The system of claim 13, wherein the respective outcome of the dynamical system in response to each of the action particles in the first set is predicted using a recurrent mixture density network (RMDN).