# US 12,109,701 B2 · Guided uncertainty-aware policy optimization: combining model-free and model-based strategies for sample-efficient learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,109,701 B2
- Document ID: US12109701B2
- Status: Granted
- Kind: B2
- Application number: 16/780,465
- Filed: 2020-02-03
- Published: 2021-05-20
- Granted: 2024-10-08
- Earliest claimed: 2019-11-20
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Jonathan Tremblay; Dieter Fox; Michelle Lee; Carlos Florensa; Nathan Donald Ratliff; Animesh Garg; Fabio Tozeto Ramos
- Web page: https://patentgrep.com/patent/US12109701B2

## CPC

- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- G05B 19/42: Recording and playback systems, i.e. in which the program is recorded from a cycle of operations, e.g. the cycle of operations being manually controlled, after which this record is played back on the same machine
- B25J 9/1661: {characterised by task planning, object-oriented languages}
- B25J 9/1664: {characterised by motion, path, trajectory planning}
- B25J 9/1697: {Vision controlled systems}
- G05B 13/027: {using neural networks only}
- G05B 13/04: involving the use of models or simulators
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 5/046: Forward inferencing; Production systems
- G06N 20/00: Machine learning
- G05B 2219/40499: Reinforcement learning algorithm
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G06N 3/063: using electronic means
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A robot is controlled using a combination of model-based and model-free control methods. In some examples, the model-based method uses a physical model of the environment around the robot to guide the robot. The physical model is oriented using a perception system such as a camera. Characteristics of the perception system may be are used to determine an uncertainty for the model. Based at least in part on this uncertainty, the system transitions from the model-based method to a model-free method where, in some embodiments, information provided directly from the perception system is used to direct the robot without reliance on the physical model.

## Claims

### Claim 1

A computer-implemented method, comprising:
dividing at least a portion of a physical model created based at least in part on information from a perception system into a plurality of regions;
generating estimates of uncertainty for the plurality of regions based at least in part on at least one uncertainty estimation provided by the perception system;
using the physical model to control a robot in any of the plurality of regions associated with any of the estimates of uncertainty that indicate the robot is unlikely to interact with its environment; and
using at least one reinforcement learning process instead of using the physical model to control the robot in any of the plurality of regions associated with any of the estimates of uncertainty that indicate the robot is likely to interact with its environment.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein:
the perception system is a stationary camera; and
the at least one reinforcement learning process controls the robot using data collected by a camera mounted on the robot.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the plurality of regions comprises a first set of regions that are associated with any of the estimates of uncertainty that indicate the robot is unlikely to interact with its environment and a second set of regions that are associated with any of the estimates of uncertainty that indicate the robot is likely to interact with its environment, and the method further comprises:
using the physical model to move the robot within at least one region in the first set of regions to at least one region in the second set of regions; and
using the at least one reinforcement learning process to control the robot to complete a task in one or more of the second set of regions.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, further comprising:
generating the physical model based at least in part on image data collected by at least one camera.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the at least one uncertainty estimation provided by the perception system comprises a nonparametric distribution of a plurality of poses of a region of the plurality of regions and an associated weights for each of the plurality of poses.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the at least one uncertainty estimation provided by the perception system comprises a parametric distribution.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein using the physical model to control the robot comprises moving the robot using a controller that uses target attractors defined by motion policies of the robot.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the reinforcement learning process is performed using an autoencoder that is trained using input from at least one camera mounted on the robot instead of relying on the physical model.

### Claim 9

A computer system comprising:
one or more processors; and
computer-readable memory storing executable instructions that, as a result of being executed by the one or more processors, cause the computer system to:
divide at least a portion of a physical model created based at least in part on information from a perception system into a plurality of regions;
generate estimates of uncertainty for the plurality of regions based at least in part on at least one uncertainty estimation provided by the perception system, a first set of the plurality of regions to comprise any of the plurality of regions associated with any of the estimates of uncertainty that indicate a robot is unlikely to interact with its environment, a second set of the plurality of regions to comprise any of the plurality of regions associated with any of the estimates of uncertainty that indicate the robot is likely to interact with its environment;
determine if the robot is positioned in at least one of the first set of regions or at least one of the second set of regions;
use the physical model to control the robot if it is determined that the robot is in at least one of the first set of regions; and
use at least one reinforcement learning process instead of using the physical model to control the robot if it is determined that the robot is in at least one of the second set of regions.

### Claim 10 (depends on 9)

The computer system of claim 9, wherein the at least one reinforcement learning process is to control the robot using data collected by a wrist-mounted camera positioned on the robot.

### Claim 11 (depends on 9)

The computer system of claim 9, wherein the instructions, as a result of being executed by the one or more processors, cause the computer system to updates the second set of regions using a result of controlling the robot in at least a portion of the plurality of regions.

### Claim 12 (depends on 11)

The computer system of claim 11, wherein the instructions, as a result of being executed by the one or more processors, cause the computer system to:
perform a task using the robot, and
use a result of the task to modify at least one of the first or second sets of regions.

### Claim 13 (depends on 9)

The computer system of claim 9, wherein the instructions, as a result of being executed by the one or more processors, cause the computer system to orient the physical model by at least using a deep object pose estimator to process data obtained by the perception system.

### Claim 14 (depends on 9)

The computer system of claim 9, wherein:
the perception system comprises a first camera; and
the at least one reinforcement learning process controls the robot using data collected by a second camera; and
the first and the second cameras are different cameras.

### Claim 15 (depends on 9)

The computer system of claim 9, wherein the perception system comprises a camera; and
the instructions, as a result of being executed by the one or more processors, cause the computer system to use image data collected by the camera to generate a plurality of possible poses that are consistent with the image data collected by the camera.

### Claim 16 (depends on 9)

The computer system of claim 9, wherein the instructions, as a result of being executed by the one or more processors, cause the computer system to cause a controller that uses target attractors defined by motion policies of the robot to move the robot within the first set of regions.

### Claim 17

A non-transitory computer-readable medium having stored thereon instructions that, which if performed by one or more processors, cause the one or more processors to at least:
partition at least a portion of a physical model into a plurality of regions;
generate estimates of uncertainty for the plurality of regions;
identifying a first set of the plurality of regions based at least in part on any of the estimates of uncertainty associated with the first set of regions;
identifying a second set of the plurality of regions based at least in part on any of the estimates of uncertainty associated with the second set of regions;
use the physical model to control a robot in any of the first set of regions; and
use at least one reinforcement learning process instead of using the physical model to control the robot in any of the second set of regions.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein:
the physical model is to be created based at least in part on information obtained from a perception system comprising at least one stationary camera; and
the at least one reinforcement learning process controls the robot using data collected by at least one camera that moves with the robot.

### Claim 19 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein the instructions, if performed by the one or more processors, further cause the one or more processors to at least:
generate the physical model based at least in part on information obtained from a perception system, and
generate the estimates of uncertainty for the plurality of regions based at least in part on at least one uncertainty estimation provided by the perception system.

### Claim 20 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein the estimates of uncertainty are determined based at least in part on a probability distribution associated with a plurality of poses of at least one of the plurality of regions.

### Claim 21 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein the instructions, if performed by the one or more processors, further cause the one or more processors to at least:
determine that the robot is outside the second set of regions; and
as a result of determining that the robot is outside the second set of regions, use the physical model to move the robot to within at least one of the second set of regions.

### Claim 22 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein the physical model is generated using data collected by a first perception system; and
the at least one reinforcement learning process is implemented using a machine-learning model trained using input from a second perception system.

### Claim 23 (depends on 22)

The non-transitory computer-readable medium of claim 22, wherein the input comprises simulated images generated by a simulation.

### Claim 24 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein the robot is an autonomous vehicle.

### Claim 25

A processor comprising:
one or more arithmetic logic units (ALUs) to perform a task by at least:
dividing at least a portion of a physical model into a plurality of regions;
identifying a first set of the plurality of regions in which to use the physical model to control a robot;
identifying a second set of the plurality of regions in which to use at least one reinforcement learning process to control the robot;
using the physical model to control of the robot's if the robot is in the first set of regions; and
using the at least one reinforcement learning process instead of using the physical model to control the robot if the robot is in the second set of regions.

### Claim 26 (depends on 25)

The processor of claim 25, wherein;
the at least one reinforcement learning process controls the robot using data collected by at least one camera; and
the at least one camera provides an in-hand view of an object to be manipulated by the robot.

### Claim 27 (depends on 25)

The processor of claim 25, wherein the ALUs are to perform the task by at least:
generating estimates of uncertainty for the plurality of regions;
using the estimates of uncertainty to identify the first set of regions and the second set of regions;
modifying one or more of the estimates of uncertainty associated with one or more of the plurality of regions based at least in part on the robot having successfully completed a robot task in the one or more of the plurality of regions; and
using the modified one or more estimates of uncertainty to modify at least one of the first set of regions or the second set of regions.

### Claim 28 (depends on 25)

The processor of claim 25, wherein information collected as a result of the at least one reinforcement learning process controlling the robot is used to acquire an improved pose for the physical model.

### Claim 29 (depends on 25)

The processor of claim 25, wherein;
the physical model is created at least in part using first data collected by at least one first camera;
the at least one reinforcement learning process controls the robot using second data collected by at least one second camera; and
the at least one first camera is different from the at least one second camera.

### Claim 30 (depends on 25)

The processor of claim 25, wherein the ALUs are to perform the task by at least:
generating a plurality of possible poses that are consistent with image data used to generate the physical model; and
use the plurality of possible poses to identify at least one of the first set of regions or the second set of regions.