# US 11,992,943 B2 · Method for optimizing a policy for a robot

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,992,943 B2
- Document ID: US11992943B2
- Status: Granted
- Kind: B2
- Application number: 17/450,794
- Filed: 2021-10-13
- Published: 2022-04-28
- Granted: 2024-05-28
- Earliest claimed: 2020-10-28
- Assignees: Robert Bosch GmbH
- Applicants: Robert Bosch GmbH
- Inventors: Lukas Froehlich; Edgar Klenske; Leonel Rozo
- Web page: https://patentgrep.com/patent/US11992943B2

## CPC

- B25J 9/1605: {Simulation of manipulator lay-out, design, modelling of manipulator}
- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- G05B 13/042: {in which a parameter or coefficient is automatically adjusted to optimise the performance}
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A method for optimizing a predefined policy for a robot, the policy being a Gaussian mixture model. The method begins with an initialization of a Gaussian process, the Gaussian process including at least one kernel k which, as an input parameter, obtains a distance that is ascertained between probability distributions, which are characterized in each case by the Gaussian mixture model and the Gaussian process, according to the probability product kernel. This is followed by an optimization of the Gaussian process in such a way that it predicts the costs as a function of the parameters of the Gaussian mixture model. This is followed by an ascertainment of optimal parameters of the Gaussian mixture model as a function of the Gaussian process, the parameters being selected, as a function of the Gaussian process, in such a way that the Gaussian process outputs the optimal cost function.

## Claims

### Claim 1

A method for optimizing a predefined policy for a robot, the policy being a Gaussian mixture model that outputs at least one partial trajectory as a function of a starting state and a target state of the robot, the method comprising the following steps:
initializing a Gaussian process that is suitable for estimating, as a function of a parameterization of the Gaussian mixture model, costs that the robot must incur in order to reach the target state, the Gaussian process including at least one kernel which obtains an input parameter as a function of a distance that is ascertained between probability distributions, which are characterized in each case by the Gaussian mixture model and the Gaussian process, according to a probability product kernel;
creating a plurality of trajectories as a function of the policy;
ascertaining the costs for each of the trajectories of the plurality of trajectories;
optimizing the Gaussian process in such a way that, for the plurality of trajectories, the Gaussian process estimates the ascertained costs as a function of used parameters of the Gaussian mixture model;
ascertaining optimal parameters for the Gaussian mixture model with the aid of the Gaussian process, so that the Gaussian process outputs optimal costs for the optimal parameters;
replacing the parameters of the Gaussian mixture model with the optimal parameters;
supplying sensor signals to the Gaussian mixture model;
outputting, by the Gaussian mixture model, a partial trajectory, using the supplied sensor signals;
ascertaining an actuation signal as a function of the partial trajectory; and
activating and controlling an actuator of the robot using the actuation signal.

### Claim 2 (depends on 1)

The method as recited in claim 1, wherein the distance is computed according to a probability product kernel as a function of an inner product (p,q) between the Gaussian mixture model (p) and the Gaussian process (q).

### Claim 3 (depends on 2)

The method as recited in claim 2, wherein the inner product is ascertained as a function of one sum over a plurality of normal distributions, each of the plurality of normal distributions including a normal distribution for each possible combination between mixture components of the Gaussian mixture model and supporting points of the Gaussian process, each of the plurality of normal distributions being parameterized, using an expected value of a particular mixture component, given an expected value of a particular supporting point, and using one sum over variances of the particular mixture components and the particular supporting points, the plurality of normal distributions being weighted in each case with weights of the particular mixture component of the Gaussian mixture model and the particular supporting points of the Gaussian process.

### Claim 4 (depends on 3)

The method as recited in claim 3, wherein the distance is a function of a square root of the inner product (p−q,p−q) of a difference between the Gaussian mixture model and the Gaussian process and the difference.

### Claim 5 (depends on 4)

The method as recited in claim 4, wherein the inner product is solved from the differences according to a second binomial formula, and individual terms (p,p−2p,q+q,q) of the solved second binomial formula are ascertained as one sum over the plurality of normal distributions.

### Claim 6 (depends on 1)

The method as recited in claim 1, wherein the kernel is a Matèrn kernel.

### Claim 7 (depends on 1)

The method as recited in claim 1, wherein the kernel is a Matèrn 5/2 kernel.

### Claim 8 (depends on 3)

The method as recited in claim 3, wherein length scales are associated with the mixture components of the Gaussian mixture model and the supporting points of the Gaussian process, the length scales scaling the distance ascertained according to a probability product kernel.

### Claim 9

A device configured to optimize a predefined policy for a robot, the policy being a Gaussian mixture model that outputs at least one partial trajectory as a function of a starting state and a target state of the robot, the device configured to:
initialize a Gaussian process that is suitable for estimating, as a function of a parameterization of the Gaussian mixture model, costs that the robot must incur in order to reach the target state, the Gaussian process including at least one kernel which obtains an input parameter as a function of a distance that is ascertained between probability distributions, which are characterized in each case by the Gaussian mixture model and the Gaussian process, according to a probability product kernel;
create a plurality of trajectories as a function of the policy;
ascertain the costs for each of the trajectories of the plurality of trajectories;
optimize the Gaussian process in such a way that, for the plurality of trajectories, the Gaussian process estimates the ascertained costs as a function of used parameters of the Gaussian mixture model;
ascertain optimal parameters for the Gaussian mixture model with the aid of the Gaussian process, so that the Gaussian process outputs optimal costs for the optimal parameters;
replace the parameters of the Gaussian mixture model with the optimal parameters;
supply sensor signals to the Gaussian mixture model;
output, by the Gaussian mixture model, a partial trajectory, using the supplied sensor signals;
ascertain an actuation signal as a function of the partial trajectory; and
activate and control an actuator of the robot using the actuation signal.

### Claim 10

A non-transitory machine-readable memory medium on which is stored a computer program for optimizing a predefined policy for a robot, the policy being a Gaussian mixture model that outputs at least one partial trajectory as a function of a starting state and a target state of the robot, the computer program, when executed by a computer, causing the computer to perform the following steps:
initializing a Gaussian process that is suitable for estimating, as a function of a parameterization of the Gaussian mixture model, costs that the robot must incur in order to reach the target state, the Gaussian process including at least one kernel which obtains an input parameter as a function of a distance that is ascertained between probability distributions, which are characterized in each case by the Gaussian mixture model and the Gaussian process, according to a probability product kernel;
creating a plurality of trajectories as a function of the policy;
ascertaining the costs for each of the trajectories of the plurality of trajectories;
optimizing the Gaussian process in such a way that, for the plurality of trajectories, the Gaussian process estimates the ascertained costs as a function of used parameters of the Gaussian mixture model;
ascertaining optimal parameters for the Gaussian mixture model with the aid of the Gaussian process, so that the Gaussian process outputs optimal costs for the optimal parameters; and
replacing the parameters of the Gaussian mixture model with the optimal parameters;
supplying sensor signals to the Gaussian mixture model;
outputting, by the Gaussian mixture model, a partial trajectory, using the supplied sensor signals;
ascertaining an actuation signal as a function of the partial trajectory; and
activating and controlling an actuator of the robot using the actuation signal.

### Claim 11 (depends on 1)

The method as recited in claim 1, wherein the robot is an at least semi-autonomous vehicle.

### Claim 12 (depends on 1)

The method as recited in claim 1, wherein the robot is a manufacturing system.