# US 2021/0276547 A1 · Multi-agent trajectory prediction

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0276547 A1
- Document ID: US20210276547A1
- Status: Published application
- Kind: A1
- Application number: 17/187,157
- Filed: 2021-02-26
- Published: 2021-09-09
- Earliest claimed: 2020-03-04
- Assignees: NEC Laboratories America, Inc.
- Applicants: NEC Laboratories America, Inc.
- Inventors: Sriram Nochur Narayanan; Buyu Liu; Ramin Moslemi; Francesco Pittaluga; Manmohan Chandraker
- Web page: https://patentgrep.com/patent/US20210276547A1

## CPC

- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- B60W 30/0953: {the prediction being responsive to vehicle dynamic parameters}
- B60W 30/0956: {the prediction being responsive to traffic or environmental parameters}
- B60W 60/0027: {using trajectory prediction for other traffic participants}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/24137: {Distances to cluster centroïds}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06V 10/82: using neural networks
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2554/4045: Intention, e.g. lane change or imminent movement
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods and systems for training a trajectory prediction model and performing a vehicle maneuver include encoding a set of training data to generate encoded training vectors, where the training data includes trajectory information for agents over time. Trajectory scenarios are simulated based on the encoded training vectors, with each simulated trajectory scenario representing one or more agents with respective agent trajectories, to generate simulated training data. A predictive neural network model is trained using the simulated training data to generate predicted trajectory scenarios based on a detected scene.

## Claims

### Claim 1

A method for training a trajectory prediction model, comprising:
encoding a set of training data to generate encoded training vectors, where the training data includes trajectory information for agents over time;
simulating a plurality of simulated trajectory scenarios based on the encoded training vectors, each simulated trajectory scenario representing one or more agents with respective agent trajectories, to generate simulated training data; and
training a predictive neural network model using the simulated training data to generate predicted trajectory scenarios based on a detected scene.

### Claim 2 (depends on 1)

The method of claim 1, wherein the predictive neural network model includes an autoencoder section and a convolutional long-short term memory section with state pooling.

### Claim 3 (depends on 2)

The method of claim 2, wherein state pooling pools previous state information from a final layer of the convolutional long-short term memory section for all of the one or more agents and initializes a next state with the previous state information.

### Claim 4 (depends on 1)

The method of claim 1, wherein at least one of the plurality of simulated trajectory scenarios includes multiple agents.

### Claim 5 (depends on 4)

The method of claim 4, wherein the predictive model generates the predicted trajectory scenarios for all of the multiple agents in each simulated trajectory scenario at once.

### Claim 6 (depends on 1)

The method of claim 1, wherein simulating the plurality of simulated trajectory scenarios includes simulating accelerating and lane change decisions for the one or more agents.

### Claim 7 (depends on 6)

The method of claim 6, wherein acceleration simulation may include generating an acceleration value as:
a(s,ν,Δν)=a(1−(ν0/ν)δ−(s/s*(ν,Δν)2)
where s is a lead agent of the one or more agents, ν is a velocity of a decision-making agent, Δν is a velocity difference between the lead agent and the decision-making agent, \delta is an exponent that influences how acceleration decreases with velocity, s is an actual distance between the lead agent and the decision-making agent, and s* is a minimum gap between the lead agent and the decision-making agent, and ν0 is a reference velocity.

### Claim 8 (depends on 6)

The method of claim 6, wherein lane change simulation may include evaluating the following inequalities:
ãc−ac+p{(ãn−an)+(ão−ao)}>Δath−Δabias
(ãn−an)>−bsafen,(ãc−ac)>−bsafec
where a is a current acceleration, ã represents a new acceleration after lane change, p is a politeness factor, Δath is a lane changing threshold, bsafe is a maximum safe deceleration, and Δabias is a bias for a particular lane, and where subscripts/superscripts c, n, and o denote a current agent, new agents, and old agents, respectively.

### Claim 9

A method for performing a vehicle maneuver, comprising:
detecting a scene with a plurality of agents;
generating a predicted trajectory scenario based on the detected scene, using a predictive model, wherein the predictive model generates trajectories for all of the plurality of agents at once; and
determining a safe maneuver based on the predicted trajectory scenario; and
controlling a vehicle to perform the safe maneuver.

### Claim 10 (depends on 9)

The method of claim 9, wherein detecting the scene includes sensing positions and relative velocities of agent vehicles using a sensor on a decision-making vehicle.

### Claim 11 (depends on 9)

The method of claim 9, wherein the safe maneuver includes acceleration information and lane change information for the vehicle.

### Claim 12 (depends on 11)

The method of claim 11, wherein controlling the vehicle includes triggering an acceleration in accordance with the acceleration information and triggering a lane change in accordance with the lane change information.

### Claim 13 (depends on 9)

The method of claim 9, wherein the predictive model includes an autoencoder section and a convolutional long-short term memory section with state pooling.

### Claim 14 (depends on 13)

The method of claim 13, wherein state pooling pools previous state information from a final layer of the convolutional long-short term memory section for all of the one or more agents and initializes a next state with the previous state information.

### Claim 15

A system for performing a vehicle maneuver, comprising:
a vehicle control system that causes the vehicle to perform a maneuver in response to an instruction;
a sensor that detects a scene with a plurality of agents;
a hardware processor; and
a memory that stores computer program code, which, when executed by the hardware processor, causes the hardware processor to:
generate a predicted trajectory scenario based on the detected scene, using a predictive model, wherein the predictive model generates trajectories for all of the plurality of agents at once;
determine a safe maneuver based on the predicted trajectory scenario; and
issue an instruction to the vehicle control system to perform the safe maneuver.

### Claim 16 (depends on 15)

The method of claim 15, wherein the sensor senses positions and relative velocities of agent vehicles.

### Claim 17 (depends on 15)

The method of claim 15, wherein the safe maneuver includes acceleration information and lane change information for the vehicle.

### Claim 18 (depends on 17)

The method of claim 17, wherein the vehicle control system triggers an acceleration in accordance with the acceleration information and triggers a lane change in accordance with the lane change information.

### Claim 19 (depends on 15)

The method of claim 15, wherein the predictive model includes an autoencoder section and a convolutional long-short term memory section with state pooling.

### Claim 20 (depends on 15)

The method of claim 15, wherein state pooling pools previous state information from a final layer of the convolutional long-short term memory section for all of the one or more agents and initializes a next state with the previous state information.
