# US 12,112,623 B1 · Method for predicting trajectory of traffic participant in complex heterogeneous environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,623 B1
- Document ID: US12112623B1
- Status: Granted
- Kind: B1
- Application number: 18/537,771
- Filed: 2023-12-12
- Published: 2024-10-10
- Granted: 2024-10-08
- Earliest claimed: 2023-04-04
- Assignees: Tongji University
- Applicants: Tongji University
- Inventors: Lin Zhang; Hong Chen; Rongjie Yu; Qiang Meng; Jinlong Hong
- Web page: https://patentgrep.com/patent/US12112623B1

## CPC

- G08G 1/0133: {for classifying traffic situation}
- B60W 60/00276: {for two or more other traffic participants}
- G08G 1/015: with provision for distinguishing between two or more types of vehicles, e.g. between motor-cars and cycles
- Y02T 10/40: Engine management systems
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- Y02T: CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION

## Abstract

Disclosed is a method for predicting a trajectory of a traffic participant in a complex heterogeneous environment, including the following steps: obtaining traffic participant information in a complex heterogeneous environment; arranging and numbering traffic participant classes based on the class information, to obtain serial numbers of the traffic participant classes; establishing a position graph, a velocity graph, an acceleration graph, and a class graph, into each of which expert experience is introduced; and capturing topological structure relationships and time dependence relationships to obtain a position hidden state, a velocity hidden state, an acceleration hidden state, and a class hidden state; classifying the position hidden state, the velocity hidden state, the acceleration hidden state, and the class hidden state to obtain a hidden state set of traffic participants; and decoding hidden states of the traffic participants separately using a corresponding decoder to obtain future trajectory predictions of the traffic participants.

## Claims

### Claim 1

A method for predicting a trajectory of a traffic participant in a complex heterogeneous environment, comprising the following steps:
obtaining traffic participant information in a complex heterogeneous environment, wherein the traffic participant information comprises position information, velocity information, acceleration information, and class information;
arranging and numbering traffic participant classes based on the class information, to obtain serial numbers of the traffic participant classes;
establishing, based on the traffic participant information, a position graph, a velocity graph, an acceleration graph, and a class graph, into each of which expert experience is introduced;
capturing topological structure relationships and time dependence relationships from the position graph, the velocity graph, the acceleration graph, and the class graph based on four parallel spatial relationship capture networks and time dependence relationship capture networks, to obtain a position hidden state, a velocity hidden state, an acceleration hidden state, and a class hidden state;
classifying the position hidden state, the velocity hidden state, the acceleration hidden state, and the class hidden state through determination by equations based on the serial numbers to obtain a hidden state set of traffic participants; and
decoding hidden states of the traffic participants separately using a multi-mode decoder based on traffic participant classes to obtain future trajectory predictions of the traffic participants;
planning trajectory for an intelligent vehicle based on the future trajectory predictions of the traffic participants.

### Claim 2 (depends on 1)

The method for predicting a trajectory of a traffic participant in a complex heterogeneous environment according to claim 1, wherein the traffic participant classes comprise pedestrian, bicycle, and vehicle.

### Claim 3 (depends on 2)

The method for predicting a trajectory of a traffic participant in a complex heterogeneous environment according to claim 2, wherein the arranging and numbering traffic participant classes based on the class information, to obtain serial numbers of the traffic participant classes specifically comprise: arranging the class information as a class sequence in order, denoted by [C1, C2 . . . Ci . . . CN], wherein Ci represents a class of the ith traffic participant; Ci=1 represents that the ith traffic participant is a pedestrian; Ci=2 represents that the ith traffic participant is a bicycle; Ci=3 represents that the ith traffic participant is a vehicle; N represents a number of traffic participants; and serial numbers corresponding to the class sequence are [1, 2, . . . i . . . N].

### Claim 4 (depends on 2)

The method for predicting a trajectory of a traffic participant in a complex heterogeneous environment according to claim 2, wherein the class graph into which the expert experience is introduced uses a traffic participant risk coefficient as a node and a risk degree as an edge, and an expert experience correction coefficient related to a transportation agent class is introduced into the edge, wherein the traffic participant risk coefficient depends on the traffic participant classes; and a relative distance between the risk degree and a traffic participant is related to an included angle of a velocity vector.

### Claim 5 (depends on 4)

The method for predicting a trajectory of a traffic participant in a complex heterogeneous environment according to claim 4, wherein the expert experience correction coefficient related to the transportation agent class is:
OAE
C
i
⁢
j
=
{
1
+
❘
"\[LeftBracketingBar]"
v
→
i
·
D
→
j
⁢
i

v
→
i

·

D
→
j
⁢
i

❘
"\[RightBracketingBar]"
,
❘
"\[LeftBracketingBar]"
acos
⁡
(
v
→
i
·
D
→
j
⁢
i

v
→
i

·

D
→
j
⁢
i

)
❘
"\[RightBracketingBar]"
⩽
π
2
0
,
others
wherein OAEcij represents the expert experience correction coefficient related to the transportation agent class; {right arrow over (ν)}i represents a velocity vector of a traffic participant i; and {right arrow over (D)}ji represents a relative position vector between the traffic participant i and a traffic participant j.

### Claim 6 (depends on 5)

The method for predicting a trajectory of a traffic participant in a complex heterogeneous environment according to claim 5, wherein the class graph is established by:
G
C
=
(
U
C
,
E
C
)
U
C
=
{
u
C
i
=
(
C
c
i
)
⁢
❘
"\[LeftBracketingBar]"
i
=
1
,
2
,
…
,
N
;
c
∈
[
1
,
2
,
3
]
}
E
C
=
{
e
C
i
⁢
j
=
OAE
C
i
⁢
j
×
1

D
→
j
⁢
i

+
1
⁢
❘
"\[LeftBracketingBar]"
i
,
j
=
1
,
2
,
…
,
N
}
wherein Gc represents the class graph; Uc is a node of the class graph, representing the traffic participant risk coefficient; c∈[1,2,3] corresponds to a pedestrian, a bicycle, or a vehicle, respectively; Ec is an edge of the class graph, representing the risk degree; and {right arrow over (ν)}j represents the velocity vector of the traffic participant j.

### Claim 7 (depends on 1)

The method for predicting a trajectory of a traffic participant in a complex heterogeneous environment according to claim 1, wherein the classifying the position hidden state, the velocity hidden state, the acceleration hidden state, and the class hidden state through determination by equations based on the serial numbers specifically comprises: performing matching on arranged features based on the serial numbers according to a position hidden state arrangement, a velocity hidden state arrangement, an acceleration hidden state arrangement, and a class hidden state arrangement output by the spatial relationship capture networks and the time dependence relationship capture networks, classifying features having a same serial number into a same class when the serial numbers in different arrangements are the same, namely when determination by equations is met, and obtaining a pedestrian hidden state set, a bicycle hidden state set, and a vehicle hidden state set based on serial number classes.

### Claim 8 (depends on 1)

The method for predicting a trajectory of a traffic participant in a complex heterogeneous environment according to claim 1, wherein the spatial relationship capture network is a graph convolutional neural network.

### Claim 9 (depends on 1)

The method for predicting a trajectory of a traffic participant in a complex heterogeneous environment according to claim 1, wherein the time dependence relationship capture network is a temporal convolutional network, a Long Short-Term Memory neural network, a recurrent neural network, or a gated neural network.

### Claim 10 (depends on 1)

The method for predicting a trajectory of a traffic participant in a complex heterogeneous environment according to claim 1, wherein the multi-mode decoder is established by using a Long Short-Term Memory neural network, a convolutional neural network, a temporal convolutional network, a recurrent neural network, or a gated neural network.