US 10,792,810 B1Granted

Artificial intelligence system for learning robotic control policies

AssigneeAmazon Technologies, Inc.

InventorsBrian C. Beckman, Leonardo Ruggiero Bachega, Brandon William Porter, Benjamin Lev Snyder, Michael Vogelsong, Corrinne Yu

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

A machine learning system builds and uses computer models for controlling robotic performance of a task. Such computer models may be first trained using feedback on computer simulations of the robot performing the task, and then refined using feedback on real-world trials of the robot performing the task. Some examples of the computer models can be trained to automatically evaluate robotic task performance and provide the feedback. This feedback can be used by a machine learning system, for example an evolution strategies system or reinforcement learning system, to generate and refine the controller.

Claims20

Dependent claims

A system comprising:

a robotic system that performs a task according to a robotic controller programmed to implement a control policy;
and
one or more processors programmed with executable instructions to at least:
implement a physics simulation engine that simulates a real world environment in which the robotic system is to perform the task, wherein the physics simulation engine runs simulations of a simulated version of the robotic system performing a simulated version of the task;
use machine learning to learn parameters of a simulated control policy based on evaluations of a first level of success at the simulated version of the task within the simulations, wherein the first level of success is determined based on whether the simulated version of the robotic system successfully performed the simulated version of the task within the simulations;
provide the parameters of the simulated control policy learned from the simulations via the machine learning to the robotic controller of the robotic system;
run trials of the robotic system performing the task in the real world environment under control of the robotic controller;
use machine learning to generate a refined control policy by refining the parameters of the simulated control policy based on evaluations of a second level of success at the task within the real world environment, wherein the second level of success is determined based on whether the robotic system successfully performed the task during the trials in the real world environment;
and
after finalizing refinement of the parameters, program the robotic controller to control the robotic system using the refined control policy to complete performance of the task.

7 more elements

A computer-implemented method comprising, as performed by at least one computing device configured to execute specific instructions:

generating a simulated environment that corresponds to a real world environment in which a robotic system is to perform a task;
running simulations of a simulated version of the robotic system performing a simulated version of the task in the simulated environment under control of a simulated control policy;
using machine learning to learn parameters of the simulated control policy based on evaluations of a first level of success at the simulated version of the task within the simulations, wherein the first level of success is determined based on whether the simulated version of the robotic system successfully performed the simulated version of the task within the simulations;
providing the parameters of the simulated control policy learned from the simulations via the machine learning to a robotic controller that controls the robotic system;
running trials of the robotic system performing the task in the real world environment under control of the robotic controller;
using machine learning to generate a refined control policy by refining the parameters of the simulated control policy based on evaluations of a second level of success at the task within the real world environment, wherein the second level of success is determined based on whether the robotic system successfully performed the task during the trials in the real world environment;
and
after finalizing refinement of the parameters, control the robotic system to complete performance of the task using the robotic controller programmed with the refined control policy.

5 more elements

A non-transitory computer storage medium storing executable code, wherein the executable code configures a computing system to perform a process comprising:

receiving data representing evaluations of simulations of a simulated version of a robotic system performing a simulated version of a task in a simulated environment under control of a simulated control policy, the simulated environment corresponding to a real world environment, the evaluations representing a first level of success at the simulated version of the task within the simulations, wherein the first level of success is based at least in part on whether the simulated version of the robotic system successfully performed the simulated version of the task within the simulations;
using machine learning to learn parameters of a control policy based on the evaluations of the simulations;
receiving data representing evaluations of trials of the robotic system performing the task in the real world environment under control of a robotic controller programmed to implement the control policy, the evaluations representing a second level of success at the task within the real world environment, wherein the second level of success is determined based on whether the robotic system successfully performed the task during the trials in the real world environment;
using machine learning to generate a refined control policy by refining the parameters of the control policy based on the evaluations of the trials;
and
providing the refined control policy to the robotic controller of the robotic system to control the robotic system to complete performance of the task.

3 more elements

Dependencies are inferred from the claim text.