# US 12,217,168 B2 · Agent trajectory prediction using vectorized inputs

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,168 B2
- Document ID: US12217168B2
- Status: Granted
- Kind: B2
- Application number: 17/099,656
- Filed: 2020-11-16
- Published: 2021-05-20
- Granted: 2025-02-04
- Earliest claimed: 2019-11-15
- Assignees: Waymo LLC
- Applicants: Waymo LLC
- Inventors: Jiyang Gao; Yi Shen; Hang Zhao; Chen Sun
- Web page: https://patentgrep.com/patent/US12217168B2

## CPC

- B60W 60/0027: {using trajectory prediction for other traffic participants}
- G06N 3/08: Learning methods
- G01C 21/3848: {Data obtained from both position sensors and additional sensors}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/048: Activation functions
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G08G 1/0112: {from the vehicle, e.g. floating car data [FCD]}
- G08G 1/0133: {for classifying traffic situation}
- G08G 1/0141: {for traffic information dissemination}
- G08G 1/04: using optical or ultrasonic detectors
- G08G 1/166: {for active traffic, e.g. moving vehicles, pedestrians, bikes}
- G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)}
- G05D 1/227: Handing over between remote control and on-board control; Handing over between remote control arrangements
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30232: Surveillance
- G06T 2207/30236: Traffic on road, railway or crossing
- G06T 2207/30241: Trajectory
- H04W 4/44: for communication between vehicles and infrastructures, e.g. vehicle-to-cloud [V2C] or vehicle-to-home [V2H]
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using vectorized inputs.

## Claims

### Claim 1

A method performed by one or more computers, the method comprising:
receiving an input including (i) data characterizing observed trajectories for each of one or more agents in an environment and (ii) map features of a map of the environment;
generating a respective polyline of each of the observed trajectories that represents the observed trajectory as a sequence of one or more vectors, comprising generating a respective vector for each of one or more time intervals during the observed trajectory, wherein each respective vector includes features of the observed trajectory during the corresponding time interval;
generating a respective polyline of each of the features of the map that represents the feature as a sequence of one or more vectors;
processing a network input comprising the (i) respective polylines of the observed trajectories and (ii) the respective polylines of each of the features of the map using an encoder neural network to generate polyline features for each of the one or more agents;
for one or more of the agents, generating a predicted trajectory for the agent from the polyline features for the agent; and
controlling a movement of a vehicle based on the trajectories predicted for the one or more agents.

### Claim 2 (depends on 1)

The method of claim 1, wherein processing the network input comprises:
for each polyline, processing the vectors in the polyline using a graph neural network to generate initial polyline features of the polyline; and
processing the initial polyline features for each of the polylines using a self-attention neural network to generate the respective polyline features for each of the one or more agents.

### Claim 3 (depends on 1)

The method of claim 1, wherein generating the predicted trajectory for the agent comprises:
processing the polyline features for the agent using a trajectory decoder neural network to generate the predicted trajectory.

### Claim 4 (depends on 1)

The method of claim 1, wherein the respective vector for each of the one or more time intervals comprises:
coordinates of a position of the agent along the trajectory at a beginning of the time interval; and
coordinates of a position of the agent along the trajectory at an end of the time interval.

### Claim 5 (depends on 1)

The method of claim 1, wherein the respective vector for each of the one or more time intervals comprises:
a timestamp identifying the time interval.

### Claim 6 (depends on 1)

The method of claim 1, wherein the respective vector for each of the one or more time intervals comprises:
a label identifying the polyline to which the vector belongs.

### Claim 7 (depends on 1)

The method of claim 1, wherein generating a respective polyline of each of the features of the map that represents the feature as a sequence of one or more vectors comprises:
generating vectors connecting a plurality of keypoints along the feature in the map.

### Claim 8 (depends on 7)

The method of claim 7, wherein each of the vectors connecting a plurality of keypoints along the feature in the map comprises:
coordinates of a position of a start of the vector in the environment; and
coordinates of a position of an end of the vector in the environment.

### Claim 9 (depends on 7)

The method of claim 7, wherein each of the vectors connecting a plurality of keypoints along the feature in the map comprises:
one or more attribute features of the map feature.

### Claim 10 (depends on 7)

The method of claim 7, wherein each of the vectors connecting a plurality of keypoints along the feature in the map comprises:
a label identifying the polyline to which the vector belongs.

### Claim 11

A system comprising one or more computers and one or more storage devices storing instructions, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
receiving an input including (i) data characterizing observed trajectories for each of one or more agents in an environment and (ii) map features of a map of the environment;
generating a respective polyline of each of the observed trajectories that represents the observed trajectory as a sequence of one or more vectors, comprising generating a respective vector for each of one or more time intervals during the observed trajectory, wherein each respective vector includes features of the observed trajectory during the corresponding time interval;
generating a respective polyline of each of the features of the map that represents the feature as a sequence of one or more vectors;
processing a network input comprising the (i) respective polylines of the observed trajectories and (ii) the respective polylines of each of the features of the map using an encoder neural network to generate polyline features for each of the one or more agents;
for one or more of the agents, generating a predicted trajectory for the agent from the polyline features for the agent; and
controlling a movement of a vehicle based on the trajectories predicted for the one or more agents.

### Claim 12 (depends on 11)

The system of claim 11, wherein processing the network input comprises:
for each polyline, processing the vectors in the polyline using a graph neural network to generate initial polyline features of the polyline; and
processing the initial polyline features for each of the polylines using a self-attention neural network to generate the respective polyline features for each of the one or more agents.

### Claim 13 (depends on 11)

The system of claim 11, wherein generating the predicted trajectory for the agent comprises:
processing the polyline features for the agent using a trajectory decoder neural network to generate the predicted trajectory.

### Claim 14 (depends on 11)

The system of claim 11, wherein the respective vector for each of the one or more time intervals comprises:
coordinates of a position of the agent along the trajectory at a beginning of the time interval; and
coordinates of a position of the agent along the trajectory at an end of the time interval.

### Claim 15 (depends on 11)

The system of claim 11, wherein the respective vector for each of the one or more time intervals comprises:
a timestamp identifying the time interval.

### Claim 16 (depends on 11)

The system of claim 11, wherein the respective vector for each of the one or more time intervals comprises:
a label identifying the polyline to which the vector belongs.

### Claim 17 (depends on 11)

The system of claim 11, wherein generating a respective polyline of each of the features of the map that represents the feature as a sequence of one or more vectors comprises:
generating vectors connecting a plurality of keypoints along the feature in the map.

### Claim 18

One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
receiving an input including (i) data characterizing observed trajectories for each of one or more agents in an environment and (ii) map features of a map of the environment;
generating a respective polyline of each of the observed trajectories that represents the observed trajectory as a sequence of one or more vectors, comprising generating a respective vector for each of one or more time intervals during the observed trajectory, wherein each respective vector includes features of the observed trajectory during the corresponding time interval;
generating a respective polyline of each of the features of the map that represents the feature as a sequence of one or more vectors;
processing a network input comprising the (i) respective polylines of the observed trajectories and (ii) the respective polylines of each of the features of the map using an encoder neural network to generate polyline features for each of the one or more agents;
for one or more of the agents, generating a predicted trajectory for the agent from the polyline features for the agent; and
controlling a movement of a vehicle based on the trajectories predicted for the one or more agents.