Back to results
US 12,008,077 B1Granted
Training action-selection neural networks from demonstrations using multiple losses
AssigneeDeepMind Technologies Limited
InventorTodd Andrew Hester
Life of this patent
Estimating the term…
Abstract
A method of training an action selection neural network to perform a demonstrated task using a supervised learning technique. The action selection neural network is configured to receive demonstration data comprising actions to perform the task and rewards received for performing the actions. The action selection neural network has auxiliary prediction task neural networks on one or more of its intermediate outputs. The action selection policy neural network is trained using multiple combined losses, concurrently with the auxiliary prediction task neural networks.
Claims20
A method of training a neural network system using demonstrations, the method comprising:
- training an action selection neural network to perform demonstrated actions of a demonstrated task,
- wherein the action selection neural network is configured to receive inputs comprising observations of an environment of the demonstrated task and to process the inputs to generate action selection outputs indicating actions to be performed, and
- during training of the action selection neural network:
- training each of one or more auxiliary prediction task neural networks, wherein each of the auxiliary prediction task neural networks is configured to receive a respective intermediate output from the action selection neural network and to generate a task-related prediction output which indicates a predicted characteristic of the demonstrated task,
- wherein training each of the one or more auxiliary prediction task neural networks comprises training each of the one or more auxiliary prediction task neural networks and the action selection neural network by backpropagating gradients determined from an auxiliary learning loss function through the one or more auxiliary prediction task neural networks and into the action selection neural network to bring the predicted characteristic closer to a corresponding subsequently-observed characteristic of the demonstrated task.
2 more elements
A system comprising:
- one or more computers;
- and
- one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of training a neural network system using demonstrations, the operations comprising:
- training an action selection neural network to perform demonstrated actions of a demonstrated task,
- wherein the action selection neural network is configured to receive inputs comprising observations of an environment of the demonstrated task and to process the inputs to generate action selection outputs indicating actions to be performed, and
- during training of the action selection neural network:
- training each of one or more auxiliary prediction task neural networks, wherein each of the auxiliary prediction task neural networks is configured to receive a respective intermediate output from the action selection neural network and to generate a task-related prediction output which indicates a predicted characteristic of the demonstrated task,
- wherein training each of the one or more auxiliary prediction task neural networks comprises training each of the one or more auxiliary prediction task neural networks and the action selection neural network by backpropagating gradients determined from an auxiliary learning loss function through the one or more auxiliary prediction task neural networks and into the action selection neural network to bring the predicted characteristic closer to a corresponding subsequently-observed characteristic of the demonstrated task.
5 more elements
One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of training a neural network system using demonstrations, the operations comprising:
- training an action selection neural network to perform demonstrated actions of a demonstrated task,
- wherein the action selection neural network is configured to receive inputs comprising observations of an environment of the demonstrated task and to process the inputs to generate action selection outputs indicating actions to be performed, and
- during training of the action selection neural network:
- training each of one or more auxiliary prediction task neural networks, wherein each of the auxiliary prediction task neural networks is configured to receive a respective intermediate output from the action selection neural network and to generate a task-related prediction output which indicates a predicted characteristic of the demonstrated task,
- wherein training each of the one or more auxiliary prediction task neural networks comprises training each of the one or more auxiliary prediction task neural networks and the action selection neural network by backpropagating gradients determined from an auxiliary learning loss function through the one or more auxiliary prediction task neural networks and into the action selection neural network to bring the predicted characteristic closer to a corresponding subsequently-observed characteristic of the demonstrated task.
2 more elements
Dependencies are inferred from the claim text.