# US 2022/0035375 A1 · Predictive motion planning system and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0035375 A1
- Document ID: US20220035375A1
- Status: Published application
- Kind: A1
- Application number: 16/940,807
- Filed: 2020-07-28
- Published: 2022-02-03
- Earliest claimed: 2020-07-28
- Assignees: Huawei Technologies Co., Ltd.
- Inventors: Kasra REZAEE; Peyman YADMELLAT
- Web page: https://patentgrep.com/patent/US20220035375A1

## CPC

- G05D 1/0221: {involving a learning process}
- B60W 60/0011: {involving control alternatives for a single driving scenario, e.g. planning several paths to avoid obstacles}
- G05B 13/027: {using neural networks only}
- G06F 11/3089: {Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents (software debugging using additional hardware using a specific debug interface G06F11/3656; performance evaluation by tracing or monitoring G06F11/3466)}
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/08: Learning methods
- G06N 3/092: Reinforcement learning
- G06F 11/3013: {where the computing system is an embedded system, i.e. a combination of hardware and software dedicated to perform a certain function in mobile devices, printers, automotive or aircraft systems (testing or monitoring of control systems or parts thereof G05B23/02)}
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods and systems for training a motion planner for an autonomous vehicle are described. A trajectory evaluator agent of the motion planner receives state data defining a current state of the autonomous vehicle and an environment at a current time step. Based on the current state, a trajectory is selected. A reward is calculated based on performance of the selected trajectory in the current state. State data is received for a next state of the autonomous vehicle and the environment at a next time step. Parameters of the trajectory evaluator agent are updated based on the current state, selected trajectory, computed reward and next state. The parameters of the trajectory evaluator agent are updated to assign an evaluation value for the selected trajectory that reflects the calculated reward and expected performance of the selected trajectory in the future states.

## Claims

### Claim 1

A system for training a motion planner for an autonomous vehicle, the system comprising a processing unit configured to execute instructions to cause the system to:
receive, as input to a trajectory evaluator agent of the motion planner, a first set of state data defining a current state of the autonomous vehicle and an environment at a current time step;
select, based on the current state, a selected trajectory;
compute reward for the selected trajectory based on performance of the selected trajectory in the current state;
receive a second set of state data defining a next state of the autonomous vehicle and the environment at a next time step; and
update parameters of the trajectory evaluator agent based on the current state, selected trajectory, computed reward and next state, the parameters of the agent of the trajectory evaluator being updated to assign an evaluation value for the selected trajectory that reflects the computed reward and expected performance of the selected trajectory in the future states.

### Claim 2 (depends on 1)

The system of claim 1, wherein the first set of state data and the second set of state data each independently includes state data encoded in the form of 2D images.

### Claim 3 (depends on 1)

The system of claim 1, wherein the parameters of the trajectory evaluator agent are updated according to the equation:
V(st,τt)←rt+γV(st+1,τt+1)
where st is the current state at the current time step t, τt is the selected trajectory, rt is the calculated reward, V(st,τt) is the evaluation value for the selected trajectory at the current time step and the current state, t+1 is the next time step, v(st+1,τt+1) is an evaluation value for the selected trajectory at the next time step and the next state, and γ is a discount factor.

### Claim 4 (depends on 1)

The system of claim 1, wherein the parameters of the trajectory evaluator are updated according to the equation:
V(st,τt)←rt+γV(st+1,TS(st+1))
where st is the current state at the current time step t, τt is the selected trajectory, rt is the computed reward, V(st,τt) is the evaluation value for the selected trajectory at the current time step and the current state, t+1 is the next time step, TS(st+1) is a next selected trajectory at the next time step, V(st+1,TS(st+1)) is an evaluation value for the next selected trajectory and the next state, and γ is a discount factor.

### Claim 5 (depends on 1)

The system of claim 1, wherein the evaluation value is generated as a set of statistical metrics defining a probability distribution of a probabilistic evaluation value.

### Claim 6 (depends on 5)

The system of claim 5, wherein the selected trajectory is selected according to a selection criteria based on one or more statistical metrics.

### Claim 7 (depends on 1)

The system of claim 1, wherein the set of data defining the candidate trajectory is a set of parameters defining the candidate trajectory according to a trajectory generation function.

### Claim 8 (depends on 7)

The system of claim 7, wherein the processing unit is configured to execute instructions to further cause the system to:
generate the selected trajectory from the set of parameters, according to the trajectory generation function.

### Claim 9 (depends on 1)

The system of claim 1, wherein the selected trajectory is defined by a set of 2D images defining waypoints of the candidate trajectory over multiple time steps.

### Claim 10

A method for training a motion planner for an autonomous vehicle, the method comprising:
receiving, as input to a trajectory evaluator of the motion planner, a first set of state data defining a current state of the autonomous vehicle and an environment at a current time step;
selecting, based on the current state, a selected trajectory;
computing a reward for the selected trajectory based on performance of the selected trajectory in the current state;
receiving a second set of state data defining a next state of the autonomous vehicle and the environment at a next time step; and
updating parameters of the trajectory evaluator agent based on the current state, selected trajectory, computed reward and next state, the parameters of the trajectory evaluator agent being updated to assign an evaluation value for the selected trajectory that reflects the computed reward and expected performance of the selected trajectory in the future states.

### Claim 11 (depends on 10)

The method of claim 10, wherein the first set of state data and the second set of state data each independently includes state data encoded in the form of 2D images.

### Claim 12 (depends on 10)

The method of claim 10, wherein the parameters of the trajectory evaluator are updated according to the equation:
V(st,τt)←rt+γV(st+1,τt+1)
where st is the current state at the current time step t, τt is the selected trajectory, rt is the calculated reward, V(st,τt) is the evaluation value for the selected trajectory at the current time step and the current state, t+1 is the next time step, v(st+1,τt+1) is an evaluation value for the selected trajectory at the next time step and the next state, and γ is a discount factor.

### Claim 13 (depends on 10)

The method of claim 10, wherein the parameters of the trajectory evaluator are updated according to the equation:
V(st,rt)←rt+γV(st+1,TS(st+1))
where st is the current state at the current time step t, τt is the selected trajectory, rt is the calculated reward, V(st,τt) is the evaluation value for the selected trajectory at the current time step and the current state, t+1 is the next time step, TS(st+1) is a next selected trajectory at the next time step, V(st+1,TS(st+1)) is an evaluation value for the next selected trajectory and the next state, and γ is a discount factor.

### Claim 14 (depends on 10)

The method of claim 10, wherein the evaluation value is generated as a set of statistical metrics defining a probability distribution of a probabilistic evaluation value.

### Claim 15 (depends on 14)

The method of claim 14, wherein the selected trajectory is selected according to a selection criteria based on one or more statistical metrics.

### Claim 16 (depends on 10)

The method of claim 10, wherein the set of data defining the candidate trajectory is a set of parameters defining the candidate trajectory according to a trajectory generation function.

### Claim 17 (depends on 16)

The method of claim 16, further comprising:
generating the selected trajectory from the set of parameters, according to the trajectory generation function.

### Claim 18 (depends on 10)

The method of claim 10, wherein the selected trajectory is defined by a set of 2D images defining waypoints of the candidate trajectory over multiple time steps.

### Claim 19

A computer-readable medium storing instructions for execution by a processing unit of a system for training a motion planner for an autonomous vehicle, the instructions when executed causing the system to:
receive, as input to a trajectory evaluator of the motion planner, a first set of state data defining a current state of the autonomous vehicle and an environment at a current time step;
select, based on the current state, a selected trajectory;
compute a reward for the selected trajectory based on performance of the selected trajectory in the current state;
receive a second set of state data defining a next state of the autonomous vehicle and the environment at a next time step; and
update parameters of the trajectory evaluator based on the current state, selected trajectory, computed reward and next state, the parameters of the trajectory evaluator being updated to assign an evaluation value for the selected trajectory that reflects the calculated reward and expected performance of the selected trajectory in the future states.

### Claim 20 (depends on 19)

The computer-readable medium of claim 19, wherein the parameters of the trajectory evaluator are updated according to the equation:
V(st,τt)←rt+γV(st+1,τt+1)
where st is the current state at the current time step t, τt is the selected trajectory, rt is the calculated reward, V(st,τt) is the evaluation value for the selected trajectory at the current time step and the current state, t+1 is the next time step, v(st+1,τt+1) is an evaluation value for the selected trajectory at the next time step and the next state, and γ is a discount factor.

### Claim 21 (depends on 19)

The computer-readable medium of claim 19, wherein the parameters of the trajectory evaluator are updated according to the equation:
v(st,τt)←rt+γV(st+1,TS(st+1))
where st is the current state at the current time step t, τt is the selected trajectory, rt is the calculated reward, V(st,τt) is the evaluation value for the selected trajectory at the current time step and the current state, t+1 is the next time step, TS(st+1) is a next selected trajectory at the next time step, V(st+1,TS(st+1)) is an evaluation value for the next selected trajectory and the next state, and γ is a discount factor.