# US 11,992,945 B2 · System and methods for training robot policies in the real world

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,992,945 B2
- Document ID: US11992945B2
- Status: Granted
- Kind: B2
- Application number: 17/094,521
- Filed: 2020-11-10
- Published: 2022-05-12
- Granted: 2024-05-28
- Earliest claimed: 2020-11-10
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Jie Tan; Sehoon Ha; Peng Xu; Sergey Levine; Zhenyu Tan
- Web page: https://patentgrep.com/patent/US11992945B2

## CPC

- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- B25J 9/162: {Mobile manipulator, movable base with manipulator arm mounted on it}
- B25J 9/1689: {Teleoperation}
- B25J 13/089: {Determining the position of the robot with reference to its environment}
- G05D 1/02: Control of position or course in two dimensions
- G05D 1/43: Control of position or course in two dimensions [2D]
- G06N 3/045: Combinations of networks
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/092: Reinforcement learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES

## Abstract

Techniques are disclosed that enable training a plurality of policy networks, each policy network corresponding to a disparate robotic training task, using a mobile robot in a real world workspace. Various implementations include selecting a training task based on comparing a pose of the mobile robot to at least one parameter of a real world training workspace. For example, the training task can be selected based on the position of a landmark, within the workspace, relative to the pose. For instance, the training task can be selected such that the selected training task moves the mobile robot towards the landmark.

## Claims

### Claim 1

A method implemented by one or more processors, the method comprising:
determining a pose of a mobile robot within a real world training workspace;
selecting, from a plurality of disparate policy networks each being fora corresponding component of locomotion, a corresponding policy network, wherein selecting the corresponding policy network is based on comparing the pose to a position of a center of the real world training workspace to determine the corresponding policy network with the corresponding component of locomotion that will move the mobile robot towards the center of the real world training workspace; and
for each of a plurality of iterations, and until one or more conditions are satisfied:
determining current state data of the mobile robot,
using the selected policy network and the corresponding current state data to determine one or more corresponding actions to move the mobile robot towards the center of the real world training workspace,
storing a corresponding training instance in association with the selected corresponding policy network, the corresponding training instance including at least the corresponding current state data and the one or more corresponding actions, and
implementing the one or more corresponding actions at the mobile robot;
updating one or more portions of the selected policy network using the training instances stored in association with the selected policy network in the iterations;
after cessation of the plurality of iterations:
determining an additional pose of the mobile robot within the real world training workspace, wherein the additional pose is distinct from the pose of the mobile robot, and wherein the additional pose of the mobile robot corresponds with an additional component of locomotion distinct from the component of locomotion of the pose of the mobile robot;
selecting, from the plurality of disparate policy networks, an additional policy network corresponding to the additional component of locomotion,
wherein selecting the additional policy network is based on comparing the additional pose to the center of the real world training workspace with the corresponding additional component of locomotion that will move the mobile robot towards the center of the real world training workspace,
wherein the additional policy network is distinct from the policy network, and
wherein the additional component of locomotion corresponding to the additional policy network is distinct from the component of locomotion corresponding to the policy network;
for each of a plurality of additional iterations, and until the one or more conditions are satisfied:
determining additional current state data of the mobile robot,
using the selected additional policy network and the corresponding additional current state data to generate one or more corresponding additional actions to move the mobile robot towards the center of the real world training workspace,
storing an additional corresponding training instance in association with the selected corresponding additional policy network, the corresponding additional training instance including at least the corresponding additional current state data and the one or more corresponding additional actions, and
implementing the one or more corresponding additional actions at the mobile robot; and
updating one or more portions of the selected additional policy network using the additional training instances stored in association with the selected additional policy network in the additional iterations.

### Claim 2 (depends on 1)

The method of claim 1, wherein selecting the corresponding policy network based on comparing the pose to the position of the center of the real world training workspace comprises:
dividing the real world workspace into a plurality of sectors, wherein each sector corresponds to a distinct policy network in the plurality of disparate policy networks;
determining the sector, of the plurality of sectors, corresponding to the position of the center of the real world training workspace; and
selecting the corresponding policy network based on the determined sector.

### Claim 3 (depends on 2)

The method of claim 2, wherein the plurality of sectors are based on the relative pose of the robot within the real world workspace.

### Claim 4 (depends on 1)

The method of claim 1, wherein at least some of the updating occurs prior to cessation of the plurality of iterations.

### Claim 5 (depends on 1)

The method of claim 1, wherein the corresponding component of locomotion, for the policy network, is a forward movement component, and wherein the corresponding component of locomotion, for the additional policy network, is one of: a backwards movement component, a left movement component, and a right movement component.

### Claim 6 (depends on 1)

The method of claim 1, wherein determining the additional pose is performed immediately following the cessation of the additional iterations, and wherein the additional pose of the mobile robot is a direct result of implementing the one or more corresponding actions.

### Claim 7 (depends on 1)

The method of claim 1, wherein determining the additional pose is performed immediately following after performing an automated recovery of the robot in response to a fall of the robot that led to the cessation of the additional iterations, and wherein the additional pose of the mobile robot is a direct result of implementing the automated recovery of the robot.

### Claim 8 (depends on 1)

The method of claim 1, wherein updating one or more portions of the selected policy network using the training instances stored in association with the selected policy network in the iterations comprises:
generating a reward based on the training instances; and
updating one or more portions of the selected policy network based on the determined reward.

### Claim 9 (depends on 1)

The method of claim 1, wherein determining the pose of the mobile robot within the real world training workspace comprises:
determining the pose of the mobile robot within the real world workspace based on one or more instances of sensor data captured via one or more sensors.

### Claim 10 (depends on 9)

The method of claim 9, wherein the one or more instances of sensor data include one or more instances of vision data captured via one or more vision sensors mounted overhead the workspace, and wherein determining the pose of the robot within the real world workspace comprises determining the pose of the mobile robot based on the one or more instances of vision data.

### Claim 11 (depends on 9)

The method of claim 9, wherein the one or more instances of sensor data include one or more instances of signal strength data transmitted via one or more wireless signal transmitters placed in the real world workspace and captured via one or more receivers of the mobile robot, and wherein determining the pose of the mobile robot within the real world workspace comprises determining the pose of the mobile robot based on the one or more instances of signal strength data.

### Claim 12

A system comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations that include:
determining a pose of a mobile robot within a real world training workspace;
selecting, from a plurality of disparate policy networks each being fora corresponding component of locomotion, a corresponding policy network, wherein selecting the corresponding policy network is based on comparing the pose to a position of a center of the real world training workspace to determine the corresponding policy network with the corresponding component of locomotion that will move the mobile robot towards the center of the real world training workspace; and
for each of a plurality of iterations, and until one or more conditions are satisfied:
determining current state data of the mobile robot,
using the selected policy network and the corresponding current state data to determine one or more corresponding actions to move the mobile robot towards the center of the real world training workspace,
storing a corresponding training instance in association with the selected corresponding policy network, the corresponding training instance including at least the corresponding current state data and the one or more corresponding actions, and
implementing the one or more corresponding actions at the mobile robot;
updating one or more portions of the selected policy network using the training instances stored in association with the selected policy network in the iterations;
after cessation of the plurality of iterations:
determining an additional pose of the mobile robot within the real world training workspace, wherein the additional pose is distinct from the pose of the mobile robot, and wherein the additional pose of the mobile robot corresponds with an additional component of locomotion distinct from the component of locomotion of the pose of the mobile robot;
selecting, from the plurality of disparate policy networks, an additional policy network corresponding to the additional component of locomotion,
wherein selecting the additional policy network is based on comparing the additional pose to the center of the real world training workspace with the corresponding additional component of locomotion that will move the mobile robot towards the center of the real world training workspace,
wherein the additional policy network is distinct from the policy network, and
wherein the additional component of locomotion corresponding to the additional policy network is distinct from the component of locomotion corresponding to the policy network;
for each of a plurality of additional iterations, and until the one or more conditions are satisfied:
determining additional current state data of the mobile robot,
using the selected additional policy network and the corresponding additional current state data to generate one or more corresponding additional actions to move the mobile robot towards the center of the real world training workspace,
storing an additional corresponding training instance in association with the selected corresponding additional policy network, the corresponding additional training instance including at least the corresponding additional current state data and the one or more corresponding additional actions, and
implementing the one or more corresponding additional actions at the mobile robot; and
updating one or more portions of the selected additional policy network using the additional training instances stored in association with the selected additional policy network in the additional iterations.

### Claim 13 (depends on 12)

The system of claim 12, wherein selecting the corresponding policy network based on comparing the pose to the position of the center of the real world training workspace comprises:
dividing the real world workspace into a plurality of sectors, wherein each sector corresponds to a distinct policy network in the plurality of disparate policy networks;
determining the sector, of the plurality of sectors, corresponding to the position of the center of the real world training workspace; and
selecting the corresponding policy network based on the determined sector.

### Claim 14 (depends on 13)

The system of claim 13, wherein the plurality of sectors are based on the relative pose of the robot within the real world workspace.

### Claim 15 (depends on 12)

The system of claim 12, wherein at least some of the updating occurs prior to cessation of the plurality of iterations.

### Claim 16 (depends on 12)

The system of claim 12, wherein the corresponding component of locomotion, for the policy network, is a forward movement component, and wherein the corresponding component of locomotion, for the additional policy network, is one of: a backwards movement component, a left movement component, and a right movement component.

### Claim 17 (depends on 12)

The system of claim 12, wherein determining the additional pose is performed immediately following the cessation of the additional iterations, and wherein the additional pose of the mobile robot is a direct result of implementing the one or more corresponding actions.