Back to results
US 11,697,205 B2Granted
Determining control policies for robots with noise-tolerant structured exploration
AssigneeGoogle Inc.
InventorsVikas Sindhwani, Atil Iscen, Krzysztof Marcin Choromanski
Life of this patent
Estimating the term…
Abstract
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for optimizing the determination of control policies for robots through the performance of simulations of robots and real-world context to determine control policy parameters.
Claims20
A method for determining a control policy for a robot, the method comprising:
- receiving a specification in the form of an objective function ƒ of a task to be performed by the robot,
- wherein ƒ is a scalar-valued objective function of a control policy, and the control policy determines, for each state s of the robot, an action u to be performed by the robot, and
- wherein the objective function ƒ represents how well the robot performs the task;
- determining the control policy for the robot to perform the task by simulating operation of the robot to determine the control policy x* by solving an optimization problem of the form
- x
- *
- =
- argmin
- x
- ∈
- ℝ
- n
-
- f
-
- (
- x
- )
- ,
- wherein x∈n is a state of the control policy encountered during simulating the operation of the robot, and wherein solving the optimization problem comprises estimating gradients of the objective function ƒ using a finite difference procedure to estimate the gradients in perturbation directions defined by rows of a balanced spinner, the balanced spinner being a two-dimensional matrix such that the absolute value of an inner product between any one of multiple rows of the two-dimensional matrix and any one of multiple columns of the two-dimensional matrix is bounded by a threshold value, and
- causing the robot to operate under control of the determined control policy x*.
18 more elements
A system comprising:
- one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
- receiving a specification in the form of an objective function ƒ of a task to be performed by the robot,
- wherein ƒ is a scalar-valued objective function of a control policy, and the control policy determines, for each state s of the robot, an action u to be performed by the robot, and
- wherein the objective function ƒ represents how well the robot performs the task;
- determining the control policy for the robot to perform the task by simulating operation of the robot to determine the control policy x* by solving an optimization problem of the form
- x
- *
- =
- argmin
- x
- ∈
- ℝ
- n
-
- f
-
- (
- x
- )
- ,
- wherein x∈n is a state of the control policy encountered during simulating the operation of the robot, and wherein solving the optimization problem comprises estimating gradients of the objective function ƒ using a finite difference procedure to estimate the gradients in perturbation directions defined by rows of a balanced spinner, the balanced spinner being a two-dimensional matrix such that the absolute value of an inner product between any one of multiple rows of the two-dimensional matrix and any one of multiple columns of the two-dimensional matrix is bounded by a threshold value, and
- causing the robot to operate under control of the determined control policy x*.
19 more elements
One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
- receiving a specification in the form of an objective function ƒ of a task to be performed by the robot,
- wherein ƒ is a scalar-valued objective function of a control policy, and the control policy determines, for each state s of the robot, an action u to be performed by the robot, and
- wherein the objective function ƒ represents how well the robot performs the task;
- and
- determining the control policy for the robot to perform the task by simulating operation of the robot to determine the control policy x* by solving an optimization problem of the form
- x
- *
- =
- argmin
- x
- ∈
- ℝ
- n
-
- f
-
- (
- x
- )
- ,
- wherein x∈n is a state of the control policy encountered during simulating the operation of the robot, and wherein solving the optimization problem comprises estimating gradients of the objective function ƒ using a finite difference procedure to estimate the gradients in perturbation directions defined by rows of a balanced spinner, the balanced spinner being a two-dimensional matrix such that the absolute value of an inner product between any one of multiple rows of the two-dimensional matrix and any one of columns of the two-dimensional matrix is bounded by a threshold value, and
- causing the robot to operate under control of the determined control policy x*.
19 more elements
Dependencies are inferred from the claim text.