# US 10,872,294 B2 · Imitation learning using a generative predecessor neural network > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,872,294 B2 - Document ID: US10872294B2 - Status: Granted - Kind: B2 - Application number: 16/585,462 - Filed: 2019-09-27 - Published: 2020-04-02 - Granted: 2020-12-22 - Earliest claimed: 2018-09-27 - Assignees: DeepMind Technologies Limited - Applicants: DeepMind Technologies Limited - Inventors: Mel Vecerik; Yannick Schroecker; Jonathan Karl Scholz - Web page: https://patentgrep.com/patent/US10872294B2 ## CPC - G06N 3/045: Combinations of networks - G06N 3/047: Probabilistic or stochastic networks - G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)} - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/0475: Generative networks - G06N 3/08: Learning methods - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/09: Supervised learning - G06N 3/092: Reinforcement learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an action selection policy neural network. In one aspect, a method comprises: obtaining an expert observation; processing the expert observation using a generative neural network system to generate a given observation-given action pair, wherein the generative neural network system has been trained to be more likely to generate a particular observation-particular action pair if performing the particular action in response to the particular observation is more likely to result in the environment later reaching the state characterized by a target observation; processing the given observation using the action selection policy neural network to generate a given action score for the given action; and adjusting the current values of the action selection policy neural network parameters to increase the given action score for the given action. ## Claims ### Claim 1 A method for training an action selection policy neural network, wherein the action selection policy neural network has a plurality of action selection policy neural network parameters, wherein the action selection policy neural network is configured to process an observation characterizing a state of an environment in accordance with values of the action selection policy neural network parameters to generate an action selection policy output, wherein the action selection policy output comprises a respective action score for each action in a predetermined set of possible actions, wherein the action selection policy output is used to select an action to be performed by an agent interacting with an environment, the method comprising: obtaining an expert observation which characterizes a state of the environment that has been classified as being relevant to accomplishing a particular task; processing the expert observation using a generative neural network system to generate a predecessor observation—predecessor action pair, wherein the predecessor observation—predecessor action pair comprises: (i) a predecessor observation characterizing a state of the environment, and (ii) a predecessor action from the predetermined set of possible actions, wherein performing the predecessor action in response to the predecessor observation is predicted to cause the environment to subsequently transition into the state characterized by the expert observation after one or more time steps; processing the predecessor observation using the action selection policy neural network in accordance with current values of the action selection policy neural network parameters to generate an action score for given predecessor action; and adjusting the current values of the action selection policy neural network parameters to increase the action score for the predecessor action which is generated by the action selection policy neural network by processing the predecessor observation. ### Claim 2 (depends on 1) The method of claim 1, wherein the generative neural network system comprises a generative autoregressive neural network. ### Claim 3 (depends on 2) The method of claim 2, wherein the generative autoregressive neural network is a masked autoregressive flow. ### Claim 4 (depends on 1) The method of claim 1, wherein processing the expert observation using the generative neural network system to generate the predecessor observation - predecessor action pair comprises: processing the expert observation using a first generative neural network to generate the predecessor observation; and processing the expert observation and the predecessor observation using a second generative neural network to generate the predecessor action. ### Claim 5 (depends on 1) The method of claim 1, further comprising: obtaining a trajectory from a replay buffer comprising a plurality of trajectories, wherein the trajectory comprises a sequence of observation - action pairs characterizing respective states of the environment while the agent interacts with the environment by performing actions selected in accordance with values of the action selection neural network parameters; selecting: (i) a particular observation - particular action pair and (ii) a future observation from the trajectory, wherein the future observation is included in an observation - action pair that is after the particular observation - particular action pair in the trajectory; processing the future observation using the generative neural network system in accordance with current values of generative neural network system parameters to generate a predicted observation predicted action pair; determining a loss based on a difference between: (i) the particular observation - particular action pair, and (ii) the predicted observation - predicted action pair; and adjusting current values of generative neural network system parameters based on the loss. ### Claim 6 (depends on 5) The method of claim 5, wherein the actions of the trajectory were selected in accordance with the current values of the action selection neural network parameters. ### Claim 7 (depends on 6) The method of claim 6, wherein selecting: (i) a particular observation - particular action pair and (ii) a future observation from the trajectory, wherein the future observation is included in an observation - action pair that is after the particular observation - particular action pair in the trajectory comprises: selecting the particular observation - particular action pair; sampling a non-negative integer value from a probability distribution; selecting the future observation from an observation - action pair in the trajectory that is after the particular observation - particular action pair in the trajectory and is separated from the particular observation - particular action pair by intervening observation - action pairs in the trajectory. ### Claim 8 (depends on 7) The method of claim 7, wherein the probability distribution is a geometric distribution. ### Claim 9 (depends on 1) The method of claim 1, wherein the expert observation is included in an expert trajectory, wherein an expert trajectory is a sequence of expert observations characterizing respective states of the environment while a given agent interacts with the environment by performing a sequence of expert actions that accomplish the particular task. ### Claim 10 (depends on 9) The method of claim 9, wherein the expert trajectory comprises an expert action corresponding to each expert observation, and further comprising: obtaining the expert action corresponding to the obtained expert observation; and adjusting the current values of the action selection policy neural network parameters to increase the particular action score for the expert action which is generated by the action selection policy neural network by processing the expert observation. ### Claim 11 (depends on 10) The method of claim 10, wherein adjusting the current values of the action selection policy neural network parameters to increase the particular action score for the expert action which is generated by the action selection policy neural network by processing the expert observation comprises: determining a gradient of a logarithm of the particular action score for the expert action with respect to the action selection policy neural network parameters; and adjusting the current values of the action selection policy neural network parameters using the gradient. ### Claim 12 (depends on 1) The method of claim 1, wherein adjusting the current values of the action selection policy neural network parameters to increase the action score for the predecessor action which is generated by the action selection policy neural network by processing the predecessor observation comprises: determining a gradient of a logarithm of the action score for the predecessor action with respect to the action selection policy neural network parameters; and adjusting the current values of the action selection policy neural network parameters using the gradient. ### Claim 13 (depends on 1) The method of claim 1, wherein the expert observation characterizes the state of the environment at a time when the agent was controlled to interact with the environment to accomplish the particular task. ### Claim 14 (depends on 1) The method of claim 1, wherein the environment is a real-world environment and the expert observation is obtained as an output of one or more sensor devices configured to sense the real-world environment. ### Claim 15 A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training an action selection policy neural network, wherein the action selection policy neural network has a plurality of action selection policy neural network parameters, wherein the action selection policy neural network is configured to process an observation characterizing a state of an environment in accordance with values of the action selection policy neural network parameters to generate an action selection policy output, wherein the action selection policy output comprises a respective action score for each action in a predetermined set of possible actions, wherein the action selection policy output is used to select an action to be performed by an agent interacting with an environment, the operations comprising: obtaining an expert observation which characterizes a state of the environment that has been classified as being relevant to accomplishing a particular task; processing the expert observation using a generative neural network system to generate a predecessor observation - predecessor action pair, wherein the predecessor observation - predecessor action pair comprises: (i) a predecessor observation characterizing a state of the environment, and (ii) a predecessor action from the predetermined set of possible actions, wherein performing the predecessor action in response to the predecessor observation is predicted to cause the environment to subsequently transition into the state characterized by the expert observation after one or more time steps; processing the predecessor observation using the action selection policy neural network in accordance with current values of the action selection policy neural network parameters to generate an action score for the predecessor action; and adjusting the current values of the action selection policy neural network parameters to increase the action score for the predecessor action which is generated by the action selection policy neural network by processing the predecessor observation. ### Claim 16 (depends on 15) The system of claim 15, wherein the generative neural network system comprises a generative autoregressive neural network. ### Claim 17 (depends on 16) The system of claim 16, wherein the generative autoregressive neural network is a masked autoregressive flow. ### Claim 18 (depends on 15) The system of claim 15, wherein processing the expert observation using the generative neural network system to generate the predecessor observation - predecessor action pair comprises: processing the expert observation using a first generative neural network to generate the predecessor observation; and processing the expert observation and the predecessor observation using a second generative neural network to generate the predecessor action. ### Claim 19 One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training an action selection policy neural network, wherein the action selection policy neural network has a plurality of action selection policy neural network parameters, wherein the action selection policy neural network is configured to process an observation characterizing a state of an environment in accordance with values of the action selection policy neural network parameters to generate an action selection policy output, wherein the action selection policy output comprises a respective action score for each action in a predetermined set of possible actions, wherein the action selection policy output is used to select an action to be performed by an agent interacting with an environment, the operations comprising: obtaining an expert observation which characterizes a state of the environment that has been classified as being relevant to accomplishing a particular task; processing the expert observation using a generative neural network system to generate a predecessor observation - predecessor action pair, wherein the predecessor observation - predecessor action pair comprises: (i) a predecessor observation characterizing a state of the environment, and (ii) a predecessor action from the predetermined set of possible actions, wherein performing the predecessor action in response to the predecessor observation is predicted to cause the environment to subsequently transition into the state characterized by the expert observation after one or more time steps; processing the predecessor observation using the action selection policy neural network in accordance with current values of the action selection policy neural network parameters to generate an action score for the given predecessor action; and adjusting the current values of the action selection policy neural network parameters to increase the action score for the predecessor action which is generated by the action selection policy neural network by processing the predecessor observation. ### Claim 20 (depends on 19) The non-transitory computer storage media of claim 19, wherein the generative neural network system comprises a generative autoregressive neural network.