# US 12,110,042 B1 · Systems and methods for generating physically realistic trajectories

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,110,042 B1
- Document ID: US12110042B1
- Status: Granted
- Kind: B1
- Application number: 17/501,480
- Filed: 2021-10-14
- Granted: 2024-10-08
- Earliest claimed: 2020-10-14
- Assignees: Aurora Operations, Inc.
- Applicants: Aurora Operations, Inc.
- Inventors: Anh Tuan Hoang; Harshayu Girase; Sai Bhargav Yalamanchi; Micol Marchetti-Bowick
- Web page: https://patentgrep.com/patent/US12110042B1

## CPC

- B60W 60/00274: {considering possible movement changes}
- B60W 40/04: Traffic conditions
- B60W 50/0097: {Predicting future conditions}
- B60W 50/06: Improving the dynamic response of the control system, e.g. improving the speed of regulation or avoiding hunting or overshoot
- B60W 60/0011: {involving control alternatives for a single driving scenario, e.g. planning several paths to avoid obstacles}
- B60W 60/0027: {using trajectory prediction for other traffic participants}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- B60W 2554/4041: Position
- B60W 2554/4042: Longitudinal speed
- B60W 2554/4044: Direction of movement, e.g. backwards
- B60W 2554/4045: Intention, e.g. lane change or imminent movement
- B60W 2556/40: High definition maps
- B60W 2556/50: of positioning data, e.g. GPS [Global Positioning System] data
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Example aspects of the present disclosure describe the generation of more realistic trajectories for a moving actor with a hybrid technique using an algorithmic trajectory shaper in a machine-learned trajectory prediction pipeline. In this manner, for example, systems and methods of the present disclosure leverage the predictive power of machine-learning approaches combined with a priori knowledge about physically realistic trajectories for a given actor as encoded in an algorithmic approach.

## Claims

### Claim 1

A computer-implemented method comprising:
(a) obtaining data indicative of an object within an environment of an autonomous vehicle;
(b) determining, using one or more machine-learned models, a plurality of motion goal paths for the object based at least in part on the data indicative of the object within the environment of the autonomous vehicle, wherein a respective motion goal path comprises a plurality of waypoints for the object;
(c) determining a plurality of acceleration profiles based on the plurality of motion goal paths for the object;
(d) determining, using a path tracking algorithm, a plurality of predicted motion trajectories of the object based on the plurality of acceleration profiles,
wherein a respective predicted motion trajectory of the object comprises a spatial-temporal trajectory that comprises a plurality of future predicted positions of the object at a plurality of respective future timesteps;
(e) determining a motion plan of the autonomous vehicle based on at least one of the predicted motion trajectories of the object, wherein the motion plan is indicative of a motion trajectory of the autonomous vehicle; and
(f) initiating a motion control of the autonomous vehicle based on the motion plan.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the data indicative of the object within the environment of the vehicle comprises:
(i) data indicative of one or more previous positions of the object, and
(ii) state data indicative of a heading of the object, a current position of the object, a velocity of the object, and an acceleration of the object.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the one or more machine-learned models comprise a graph neural network, and wherein (b) comprises:
generating, using the graph neural network, a graph for the object based on the plurality of motion goal paths, wherein the graph is encoded with:
(i) a plurality of goal nodes associated with the plurality of motion goal paths, and
(ii) an object node associated with a current state and a past state of the object.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein the one or more machine-learned models comprises one or more convolutional neural networks and a recurrent neural network, wherein the plurality of goal nodes are encoded using the one or more convolutional neural networks and the object node is encoded using the recurrent neural network.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein (c) comprises:
generating, using one or more machine-learned temporal prediction models, multiple acceleration profiles for each of the plurality of motion goal paths, a respective acceleration profile describing a predicted acceleration of the object along an associated motion goal path.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, further comprising:
obtaining map information associated with the environment of the autonomous vehicle, and wherein the plurality of motion goal paths comprise: (i) at least one map-based goal that is based on the map information associated with the environment of the autonomous vehicle, and (ii) at least one map-free goal that is based on a heading of the object.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein (d) comprises:
identifying a motion goal path of the plurality of motion goal paths for the object;
determining a goal point on the motion goal path and a control point for the object; and
determining a curvature of a predicted motion trajectory based on the goal point, the control point for the object, and an object dynamics constraint associated with a possible curvature that is executable by the object.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein (d) further comprises:
updating a state of the object based on at least one of the plurality of acceleration profiles.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the one or more machine-learned models are trained in an end-to-end manner based on a total loss comprising a mode classification loss and a trajectory error loss.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein:
the mode classification loss corresponds to one or more probabilities determined for the plurality of predicted motion trajectories; and
the trajectory error loss corresponds to one or more distance metrics determined for the plurality of predicted motion trajectories.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, further comprising:
evaluating the predicted motion trajectories based on at least one of:
(i) a lateral speed violation threshold,
(ii) a centripetal acceleration violation threshold, or
(iii) a traversal acceleration violation threshold.

### Claim 12

An autonomous vehicle control system comprising:
one or more processors; and
one or more computer-readable medium storing instructions that when executed by the one or more processors cause the autonomous vehicle control system to perform operations, the operations comprising:
(a) obtaining data indicative of an object within an environment of an autonomous vehicle;
(b) obtaining map information associated with the environment of the autonomous vehicle;
(c) determining, using one or more machine-learned models, a plurality of motion goal paths for the object based at least in part on the data indicative of the object within the environment of the autonomous vehicle and the map information associated with the environment of the autonomous vehicle;
(d) determining a plurality of acceleration profiles based on the plurality of motion goal paths for the object;
(e) determining, using a path tracking algorithm, a plurality of predicted motion trajectories of the object based on the plurality of acceleration profiles,
wherein a respective predicted motion trajectory of the object comprises a spatial-temporal trajectory that comprises a plurality of future predicted positions of the objects at a plurality of respective future timesteps;
(f) determining a motion plan of the autonomous vehicle based on at least one of the predicted motion trajectories of the object, wherein the motion plan is indicative of a motion trajectory of the autonomous vehicle; and
(g) initiating a motion control of the autonomous vehicle based on the motion plan.

### Claim 13 (depends on 12)

The autonomous vehicle control system of claim 12, wherein (d) comprises:
generating, using one or more machine-learned temporal prediction models, multiple acceleration profiles for each of the plurality of motion goal paths.

### Claim 14 (depends on 12)

The autonomous vehicle control system of claim 12, wherein (c) comprises:
encoding, using the one or more machine-learned models, a graph for the object based on the plurality of motion goal paths.

### Claim 15 (depends on 12)

The autonomous vehicle control system of claim 12, wherein (e) comprises:
identifying a motion goal path of the plurality of motion goal paths for the object;
determining a goal point on the motion goal path and a control point for the object; and
determining a curvature of a predicted motion trajectory based on the goal point, the control point for the object, and an object dynamics constraint associated with a possible curvature that is executable by the object.

### Claim 16 (depends on 12)

The autonomous vehicle control system of claim 12, further comprising:
determining a motion plan of the autonomous vehicle based on at least one of the predicted motion trajectories of the object, wherein the motion plan is indicative of a motion trajectory of the autonomous vehicle.

### Claim 17

An autonomous vehicle comprising:
one or more processors; and
one or more computer-readable medium storing instructions that when executed by the one or more processors cause the autonomous vehicle to perform operations, the operations comprising:
(a) obtaining data indicative of an object within an environment of an autonomous vehicle;
(b) determining, using one or more machine-learned models, a plurality of motion goal paths for the object based at least in part on the data indicative of the object within the environment of the autonomous vehicle, wherein a respective motion goal path comprises a plurality of waypoints for the object;
(c) determining a plurality of acceleration profiles based on the plurality of motion goal paths for the object;
(d) determining, using a path tracking algorithm, a plurality of predicted motion trajectories of the object based on the plurality of acceleration profiles,
wherein a respective predicted motion trajectory of the object comprises a spatial-temporal trajectory that comprises a plurality of future predicted positions of the object at a plurality of respective future timesteps;
(e) determining a motion plan of the autonomous vehicle based on at least one of the predicted motion trajectories of the object, wherein the motion plan is indicative of a motion trajectory of the autonomous vehicle; and
(f) initiating a motion control of the autonomous vehicle based on the motion plan.

### Claim 18 (depends on 17)

The autonomous vehicle of claim 17, wherein the autonomous vehicle comprises one or more sensors configured to collect sensor data associated with the environment of the autonomous vehicle, and wherein the data indicative of the object within the environment of the autonomous vehicle is based on the sensor data.

### Claim 19 (depends on 17)

The autonomous vehicle of claim 17, wherein the one or more machine-learned models comprise a graph neural network, and wherein (b) comprises:
generating, using the graph neural network, a graph for the object based on the plurality of motion goal paths, wherein the graph is encoded with: (i) a plurality of goal nodes associated with the plurality of motion goal paths, and (ii) an object node associated with a current state and a past state of the object.

### Claim 20 (depends on 17)

The autonomous vehicle of claim 17, the operations further comprising:
obtaining map information associated with the environment of the autonomous vehicle, and wherein the plurality of motion goal paths comprise:
(i) at least one map-based goal that is based on the map information associated with the environment of the autonomous vehicle, and
(ii) at least one map-free goal that is based on a heading of the object.
