# US 2021/0341904 A1 · Device and method for controlling a robot

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0341904 A1
- Document ID: US20210341904A1
- Status: Published application
- Kind: A1
- Application number: 17/231,786
- Filed: 2021-04-15
- Published: 2021-11-04
- Earliest claimed: 2020-04-30
- Assignees: Robert Bosch GmbH
- Applicants: Robert Bosch GmbH
- Inventors: Jan Guenter Woehlke; Felix Schmitt; Herke Van Hoof
- Web page: https://patentgrep.com/patent/US20210341904A1

## CPC

- G05B 19/4155: characterised by program execution, i.e. part program or machine function execution, e.g. selection of a program
- G05D 1/0274: {using mapping information stored in a memory device (navigation using map-matching G01C21/30)}
- G05D 1/0214: {in accordance with safety or protection criteria, e.g. avoiding hazardous areas (monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13)}
- G05D 1/0221: {involving a learning process}
- G05D 1/617: Safety or protection, e.g. defining protection zones around obstacles or avoiding hazards (arrangements for controlling the position or course of two or more vehicles for avoiding collisions therebetween G05D1/693; arrangements for reacting to or preventing system or operator failure G05D1/80)
- G06N 3/045: Combinations of networks
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G05B 13/029: {using neural networks and expert systems}
- G05B 2219/33002: Artificial intelligence AI, expert, knowledge, rule based system KBS
- G05B 2219/39271: Ann artificial neural network, ffw-nn, feedforward neural network
- G05B 2219/50391: Robot
- G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)}
- G05D 1/228: Command input arrangements located on-board unmanned vehicles
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method for controlling a robot. The method includes receiving an indication of a target configuration to be reached from an initial configuration of the robot, determining a coarse-scale value map by value iteration, starting from an initial coarse-scale state and until the robot reaches the target configuration or a maximum number of fine-scale states has been reached, determining a fine-scale sub-goal from the coarse-scale value map, performing, by an actuator of the robot, fine-scale control actions to reach the determined fine-scale sub-goal and obtaining sensor data to determine the fine-scale states reached, starting from a current fine-scale state of the robot and until the robot reaches the determined fine-scale sub-goal, the robot transitions to a different coarse-scale state, or a maximum sequence length of the sequence of fine-scale states has been reached and determining the next coarse-scale state.

## Claims

### Claim 1

A method for controlling a robot, comprising the following steps:
receiving an indication of a target configuration to be reached from an initial configuration of the robot;
determining a coarse-scale value map by value iteration, wherein transition probabilities of the coarse-scale value map are determined using a transition probability model mapping coarse-scale states and coarse-scale actions to transition probabilities for coarse-scale states; and
for each coarse-scale state of a sequence of coarse-scale states of the robot, starting from an initial coarse-scale state determined from the initial configuration of the robot and until the robot reaches the target configuration or a maximum number of fine-scale states has been reached:
determining a fine-scale sub-goal from the coarse-scale value map,
performing, by an actuator of the robot, fine-scale control actions to reach the determined fine-scale sub-goal and obtaining sensor data to determine the fine-scale states reached as a result of performing the fine-scale control actions for each fine-scale state of a resulting sequence of fine-scale states of the robot, starting from a current fine-scale state of the robot and until the robot reaches the determined fine-scale sub-goal, the robot transitions to a different coarse-scale state, or a maximum sequence length of the sequence of fine-scale states has been reached, and
determining a next coarse-scale state of the sequence of coarse-scale states from a last fine-scale state of the sequence of fine-scale states.

### Claim 2 (depends on 1)

The method of claim 1, wherein the transition probability model is a model trainable by supervised learning.

### Claim 3 (depends on 1)

The method of claim 1, wherein the transition probability model is a neural network.

### Claim 4

A robot controller configured to control a robot, the robot controller configured to:
receive an indication of a target configuration to be reached from an initial configuration of the robot;
determine a coarse-scale value map by value iteration, wherein transition probabilities of the coarse-scale value map are determined using a transition probability model mapping coarse-scale states and coarse-scale actions to transition probabilities for coarse-scale states; and
for each coarse-scale state of a sequence of coarse-scale states of the robot, starting from an initial coarse-scale state determined from the initial configuration of the robot and until the robot reaches the target configuration or a maximum number of fine-scale states has been reached:
determine a fine-scale sub-goal from the coarse-scale value map,
cause an actuator of the robot to perform fine-scale control actions to reach the determined fine-scale sub-goal and obtaining sensor data to determine the fine-scale states reached as a result of performing the fine-scale control actions for each fine-scale state of a resulting sequence of fine-scale states of the robot, starting from a current fine-scale state of the robot and until the robot reaches the determined fine-scale sub-goal, the robot transitions to a different coarse-scale state, or a maximum sequence length of the sequence of fine-scale states has been reached, and
determine a next coarse-scale state of the sequence of coarse-scale states from a last fine-scale state of the sequence of fine-scale states.

### Claim 5

A method for training a robot controller, the robot controller configured to: (i) receive an indication of a target configuration to be reached from an initial configuration of the robot, (ii) determine a coarse-scale value map by value iteration, wherein transition probabilities of the coarse-scale value map are determined using a transition probability model mapping coarse-scale states and coarse-scale actions to transition probabilities for coarse-scale states, and (iii) for each coarse-scale state of a sequence of coarse-scale states of the robot, starting from an initial coarse-scale state determined from the initial configuration of the robot and until the robot reaches the target configuration or a maximum number of fine-scale states has been reached: (a) determine a fine-scale sub-goal from the coarse-scale value map, (b) cause an actuator of the robot to perform fine-scale control actions to reach the determined fine-scale sub-goal and obtaining sensor data to determine the fine-scale states reached as a result of performing the fine-scale control actions for each fine-scale state of a resulting sequence of fine-scale states of the robot, starting from a current fine-scale state of the robot and until the robot reaches the determined fine-scale sub-goal, the robot transitions to a different coarse-scale state, or a maximum sequence length of the sequence of fine-scale states has been reached, and (c) determine a next coarse-scale state of the sequence of coarse-scale states from a last fine-scale state of the sequence of fine-scale states, the method comprising:
training the transition probability model using supervised learning and including training a fine-scale control policy via reinforcement learning that determines the fine-scale control actions to reach a sub-goal.

### Claim 6 (depends on 5)

The method as recited in claim 5, further comprising training the transition probability model by maximizing a probability of the transition probability model to predict a coarse-scale state reached by fine-scale actions performed to reach a sub-goal.

### Claim 7

A non-transitory computer-readable medium on which is stored a computer program including instructions for controlling a robot, the computer program, when executed by a computer, causing the computer to perform the following steps:
receiving an indication of a target configuration to be reached from an initial configuration of the robot;
determining a coarse-scale value map by value iteration, wherein transition probabilities of the coarse-scale value map are determined using a transition probability model mapping coarse-scale states and coarse-scale actions to transition probabilities for coarse-scale states; and
for each coarse-scale state of a sequence of coarse-scale states of the robot, starting from an initial coarse-scale state determined from the initial configuration of the robot and until the robot reaches the target configuration or a maximum number of fine-scale states has been reached:
determining a fine-scale sub-goal from the coarse-scale value map,
causing an actuator of the robot to perform fine-scale control actions to reach the determined fine-scale sub-goal and obtaining sensor data to determine the fine-scale states reached as a result of performing the fine-scale control actions for each fine-scale state of a resulting sequence of fine-scale states of the robot, starting from a current fine-scale state of the robot and until the robot reaches the determined fine-scale sub-goal, the robot transitions to a different coarse-scale state, or a maximum sequence length of the sequence of fine-scale states has been reached, and
determining a next coarse-scale state of the sequence of coarse-scale states from a last fine-scale state of the sequence of fine-scale states.
