Back to results
US 12,704,841 B2Granted
Deep reinforcement learning-based techniques for end to end robot navigation
AssigneeGoogle Inc.
InventorsAleksandra Faust, Hao-tien Chiang, Anthony Francis, Marek Fiser
Life of this patent
Estimating the term…
Abstract
Using reinforcement learning to train a policy network that can be utilized, for example, by a robot in performing robot navigation and/or other robotic tasks. Various implementations relate to techniques for automatically learning a reward function for training of a policy network through reinforcement learning, and automatically learning a neural network architecture for the policy network.
Claims20
A method of generating, via reinforcement learning, a trained policy network, the method implemented by one or more processors, and the method comprising:
- determining a given set of reward weights to use in generating parameterized atomic rewards to be processed by a critic network during training of the trained policy network, determining the given set of reward weights comprising:
- selecting a given actor network and a given critic network;
- selecting a plurality of sets of reward weights, including the given set of reward weights;
- performing a plurality of reward-shaping reinforcement learning trials, each using a respective instance of the given actor network and the given critic network, and using a respective one of the sets of reward weights, wherein performing each of the reward-shaping reinforcement learning trials comprises:
- training the respective instance of the given actor network and the given critic network using reinforcement learning based on training episodes, including using the respective one of the sets of reward weights in generating respective parameterized atomic rewards to be processed by the respective instance of the given critic network during the training;
- and
- evaluating, after the training, each of the respective instances of the given actor network to generate a respective evaluation metric;
- and
- selecting the given set of reward weights, from the plurality of sets of reward weights, based on the respective evaluation metric for the respective instance of the given actor network, trained using the given set of reward weights;
- selecting a given actor network and critic network pair, affer selecting the given set of reward weights, selecting the given actor network and critic network pair comprising:
- generating a plurality of actor network and critic network pairs, each of the actor network and critic network pairs having unique hyperparameters;
- performing a plurality of network-shaping reinforcement learning trials, each using the selected given set of reward weights, and using a respective pair of the actor network and critic network pairs, wherein using the selected given set of reward weights is responsive to selecting the given set of reward weights;
- evaluating, after the network-shaping reinforcement learning trials, the actor network of each of the actor network and critic network pairs, to generate a respective additional evaluation metric, the actor network and critic network pairs, for which the respective additional evaluation metric is generated, including a first pair having first hyperparameters and a second pair having second hyperparameters;
- and
- selecting the given actor network and critic network pair, from the actor network and critic network pairs, based on the respective additional evaluation metric for the actor network of the given actor network and critic network pair;
- and
- in response to selecting the given actor network and critic network pair, using the selected given actor network and critic network pair and the selected given set of reward weights in generating the trained policy network, wherein the trained policy network comprises the actor network, of the selected given actor network and critic network pair, after reinforcement learning based on the selected given set of reward weights.
14 more elements
A method of generating, via reinforcement learning, a trained policy network, the method implemented by one or more processors, and the method comprising:
- performing multiple instances of reinforcement learning on each of a plurality of separate instances of the same actor and critic network, each of the multiple instances of reinforcement learning utilizing a separate set of reward weights to be processed by the critic network during the reinforcement learning;
- selecting, from the sets of reward weights, a single set of reward weights to use as fixed reward weights for the trained policy network, wherein selecting the single set of reward weights is based on evaluation of the separate instances of the same actor and critic network;
- in response to selecting the single set of reward weights, using the single set of reward weights in performing reinforcement learning on each of a plurality of separate actor and critic network pairs, each of the actor and critic network pairs having neural network architectures that are distinct from other neural network architectures of other actor and critic network pairs;
- selecting, from the actor and critic network pairs and based on a respective evaluation metric for each of the actor and critic network pairs, a single actor and critic network pair;
- using the single actor and critic network pair and the selected single set of reward weights in performing further reinforcement learning to train the actor network of the single actor and critic network pair;
- and
- after performing the reinforcement learning, using the trained actor network as the trained policy network.
4 more elements
Dependencies are inferred from the claim text.