# US 12,008,077 B1 · Training action-selection neural networks from demonstrations using multiple losses > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,077 B1 - Document ID: US12008077B1 - Status: Granted - Kind: B1 - Application number: 18/120,912 - Filed: 2023-03-13 - Granted: 2024-06-11 - Earliest claimed: 2017-10-27 - Assignees: DeepMind Technologies Limited - Applicants: DeepMind Technologies Limited - Inventors: Todd Andrew Hester - Web page: https://patentgrep.com/patent/US12008077B1 ## CPC - G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade} - G06V 10/764: using classification, e.g. of video objects - G06F 17/11: for solving equations {, e.g. nonlinear equations, general mathematical optimization problems (optimization specially adapted for a specific administrative, business or logistic context G06Q10/04)} - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06N 3/092: Reinforcement learning - G06V 10/82: using neural networks - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A method of training an action selection neural network to perform a demonstrated task using a supervised learning technique. The action selection neural network is configured to receive demonstration data comprising actions to perform the task and rewards received for performing the actions. The action selection neural network has auxiliary prediction task neural networks on one or more of its intermediate outputs. The action selection policy neural network is trained using multiple combined losses, concurrently with the auxiliary prediction task neural networks. ## Claims ### Claim 1 A method of training a neural network system using demonstrations, the method comprising: training an action selection neural network to perform demonstrated actions of a demonstrated task, wherein the action selection neural network is configured to receive inputs comprising observations of an environment of the demonstrated task and to process the inputs to generate action selection outputs indicating actions to be performed, and during training of the action selection neural network: training each of one or more auxiliary prediction task neural networks, wherein each of the auxiliary prediction task neural networks is configured to receive a respective intermediate output from the action selection neural network and to generate a task-related prediction output which indicates a predicted characteristic of the demonstrated task, wherein training each of the one or more auxiliary prediction task neural networks comprises training each of the one or more auxiliary prediction task neural networks and the action selection neural network by backpropagating gradients determined from an auxiliary learning loss function through the one or more auxiliary prediction task neural networks and into the action selection neural network to bring the predicted characteristic closer to a corresponding subsequently-observed characteristic of the demonstrated task. ### Claim 2 (depends on 1) A method as claimed in claim 1, further comprising, during training of the action selection neural network: training the action selection neural network to select actions to be performed by an agent interacting with the environment to perform the demonstrated task using a reinforcement learning technique. ### Claim 3 (depends on 1) A method as claimed in claim 1 wherein the one or more auxiliary prediction task neural networks include an auxiliary prediction task neural network to predict a demonstrated action at a subsequent observation to a current observation, and wherein the predicted characteristic comprises a predicted demonstrated action. ### Claim 4 (depends on 1) A method as claimed in claim 1 wherein training the action selection neural network to perform the demonstrated task comprises training the action selection neural network using both a supervised learning technique and a reinforcement learning technique. ### Claim 5 (depends on 1) A method as claimed in claim 1 wherein the one or more auxiliary prediction task neural networks include an auxiliary prediction task neural network to predict one or more Q-values. ### Claim 6 (depends on 5) A method as claimed in claim 5 wherein the one or more Q-values comprise a time-discounted Q-value characterizing a future state of the environment. ### Claim 7 (depends on 1) A method as claimed in claim 1 wherein the one or more auxiliary prediction task neural networks include an auxiliary prediction task neural network to predict a reward from the environment at a subsequent observation to a current observation. ### Claim 8 (depends on 1) A method as claimed in claim 1 wherein the action selection neural network has a policy output to determine actions to be performed by an agent once trained, wherein the policy output defines, each possible action, a probability distribution over a set of possible returns, and wherein training the action selection neural network comprises estimating the probability distribution over the set of possible returns for each of the possible actions. ### Claim 9 (depends on 1) A method as claimed in claim 1 wherein the one or more auxiliary prediction task neural networks include an auxiliary prediction task neural network to predict termination of the demonstrated task. ### Claim 10 (depends on 1) A method as claimed in claim 1 wherein one or both of action selection network parameters and auxiliary prediction task network parameters comprise neural network weights including noise characterizing parameters, and wherein training the action selection neural network further comprises adjusting values of the noise characterizing parameters. ### Claim 11 (depends on 1) A method as claimed in claim 1 further comprising using the trained neural network system to perform the demonstrated task. ### Claim 12 (depends on 11) A method as claimed in claim 11 wherein the trained neural network system learns solely from the demonstrated task prior to using the trained neural network system to perform the demonstrated task. ### Claim 13 (depends on 1) A method as claimed in claim 1 comprising controlling or manipulating a robot or other mechanical agent to provide a demonstration of a task, and training the neural network system using the demonstration to control the robot or other mechanical agent to perform the task. ### Claim 14 A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of training a neural network system using demonstrations, the operations comprising: training an action selection neural network to perform demonstrated actions of a demonstrated task, wherein the action selection neural network is configured to receive inputs comprising observations of an environment of the demonstrated task and to process the inputs to generate action selection outputs indicating actions to be performed, and during training of the action selection neural network: training each of one or more auxiliary prediction task neural networks, wherein each of the auxiliary prediction task neural networks is configured to receive a respective intermediate output from the action selection neural network and to generate a task-related prediction output which indicates a predicted characteristic of the demonstrated task, wherein training each of the one or more auxiliary prediction task neural networks comprises training each of the one or more auxiliary prediction task neural networks and the action selection neural network by backpropagating gradients determined from an auxiliary learning loss function through the one or more auxiliary prediction task neural networks and into the action selection neural network to bring the predicted characteristic closer to a corresponding subsequently-observed characteristic of the demonstrated task. ### Claim 15 (depends on 14) A system as claimed in claim 14, wherein the operations further comprise, during training of the action selection neural network: training the action selection neural network to select actions to be performed by an agent interacting with the environment to perform the demonstrated task using a reinforcement learning technique. ### Claim 16 (depends on 14) A system as claimed in claim 14 wherein the one or more auxiliary prediction task neural networks include an auxiliary prediction task neural network to predict a demonstrated action at a subsequent observation to a current observation, and wherein the predicted characteristic comprises a predicted demonstrated action. ### Claim 17 (depends on 14) A system as claimed in claim 14 wherein training the action selection neural network to perform the demonstrated task comprises training the action selection neural network using both a supervised learning technique and a reinforcement learning technique. ### Claim 18 (depends on 14) A system as claimed in claim 14 wherein the one or more auxiliary prediction task neural networks include an auxiliary prediction task neural network to predict one or more Q-values. ### Claim 19 (depends on 18) A system as claimed in claim 18 wherein the one or more Q-values comprise a time-discounted Q-value characterizing a future state of the environment. ### Claim 20 One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of training a neural network system using demonstrations, the operations comprising: training an action selection neural network to perform demonstrated actions of a demonstrated task, wherein the action selection neural network is configured to receive inputs comprising observations of an environment of the demonstrated task and to process the inputs to generate action selection outputs indicating actions to be performed, and during training of the action selection neural network: training each of one or more auxiliary prediction task neural networks, wherein each of the auxiliary prediction task neural networks is configured to receive a respective intermediate output from the action selection neural network and to generate a task-related prediction output which indicates a predicted characteristic of the demonstrated task, wherein training each of the one or more auxiliary prediction task neural networks comprises training each of the one or more auxiliary prediction task neural networks and the action selection neural network by backpropagating gradients determined from an auxiliary learning loss function through the one or more auxiliary prediction task neural networks and into the action selection neural network to bring the predicted characteristic closer to a corresponding subsequently-observed characteristic of the demonstrated task.