# US 2023/0410659 A1 · Method for predicting pedestrian crossing behavior for intersection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410659 A1
- Document ID: US20230410659A1
- Status: Published application
- Kind: A1
- Application number: 18/249,715
- Filed: 2021-04-12
- Published: 2023-12-21
- Earliest claimed: 2020-11-26
- Assignees: Southeast University
- Applicants: Southeast University
- Inventors: Xu LI; Jinchao HU; Weiming HU
- Web page: https://patentgrep.com/patent/US20230410659A1

## CPC

- G08G 1/166: {for active traffic, e.g. moving vehicles, pedestrians, bikes}
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06V 40/25: {Recognition of walking or running movements, e.g. gait recognition}
- G08G 1/052: with provision for determining speed or overspeed {(speed measuring in general G01P)}
- Y02T 10/40: Engine management systems
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- Y02T: CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION

## Abstract

A method for predicting a pedestrian crossing behavior for an intersection includes the following steps: step 1: designing an immediate reward function; step 2: establishing a fully convolutional neural network-long-short term memory network (FCN-LSTM) model to predict a motion reward function; step 3: training the fully convolutional neural network-long-short term memory network (FCN-LSTM) model based on reinforcement learning; and step 4: predicting the pedestrian crossing behavior and performing hazard early-warning. The technical solution does not require establishment of a complex pedestrian movement model or preparation of massive labeled data sets, achieves autonomous learning of pedestrian crossing behavior features at the intersection, predicts their walking, stopping, running and other behaviors, especially predicts the pedestrian crossing behavior when inducing hazards such as pedestrian-vehicle collision and scratch in real time, and performs hazard early-warning on crossing pedestrians and passing vehicles.

## Claims

### Claim 1

A method for predicting a pedestrian crossing behavior for an intersection, comprising the following steps:
step 1: designing an immediate reward function;
step 2: establishing a fully convolutional neural network-long-short term memory network (FCN-LSTM) model to predict a motion reward function;
step 3: training the fully convolutional neural network-long-short term memory network (FCN-LSTM) model based on reinforcement learning; and
step 4: predicting the pedestrian crossing behavior and performing hazard early-warning.

### Claim 2 (depends on 1)

The method for predicting the pedestrian crossing behavior for the intersection according to claim 1, wherein step 1: designing the immediate reward function, is specifically as follows:
taking a modified time-to-collision (MTTC) currently detected by roadside millimeter wave radar as an immediate reward rt of a status, TTC only considering that a velocity of a vehicle behind is higher than a velocity of a vehicle ahead during defining of vehicle collision, and defining a modified time-to-collision (MTTC) that considers a relative position, relative velocity, and relative acceleration between a vehicle and a pedestrian:
r
t
=
M
⁢
T
⁢
T
⁢
C
t
=
Δ
⁢
V
t
±
Δ
⁢
V
t
2
+
2
⁢
Δ
⁢
A
t
⁢
Δ
⁢
X
t
Δ
⁢
A
t
(
1
)
wherein ΔXt represents the relative position, ΔVt represents the relative velocity, ΔAt, represents the relative acceleration, and according to positive and negative conditions of ΔXt, ΔVt, and ΔAt, and by making MTTCt≥0, ±in formula (1) is taken; and
in a case that a plurality of pedestrians or a plurality of vehicles are detected within a certain status of the intersection, the MTTC between each pedestrian and all the vehicles is calculated according to formula (1), and a minimum MTTC is taken as an immediate reward rt for a current status of the pedestrian.

### Claim 3 (depends on 2)

The method for predicting the pedestrian crossing behavior for the intersection according to claim 2, wherein step 2: establishing the fully convolutional neural network-long-short term memory network (FCN-LSTM) model to predict the motion reward function, is specifically as follows:
utilizing, in consideration of interdependence between pedestrian behaviors, which is manifested as spatial interdependence and mutual constraint of pedestrians, a fully convolutional neural network (FCN) to realize semantic segmentation, separating pedestrians in an input image from a background, and preserving spatial information of the pedestrians in the input image; and in addition, utilizing, in consideration of temporal continuity of the pedestrian behaviors, a long-short term memory network (LSTM) to make use of forward behavior information of the pedestrians, and establishing the fully convolutional neural network-long-short term memory network (FCN-LSTM) model to predict a reward function value of the pedestrian behaviors, that is, inputting a pedestrian crossing image at the intersection taken by a roadside camera to the FCN-LSTM model, and outputting, by the FCN-LSTM model, reward function values corresponding to three discrete behaviors of walking, running, and stopping, wherein a specific structure of the FCN-LSTM model is as follows:
1) a standard convolutional layer 1_1: an input of a first layer is an original image with pixels of Q×Q′ and a channel number of 3, 96 11×11 convolutional kernels are used to be convolved with the original input image, a step length is 4, and edges are not expanded; rectified linear units (ReLU) serve as an activation function of neurons, and after ReLU activation, a feature map with a dimension of
Q
4
×
Q
′
4
×
9
⁢
6
is output;
a local response normalization layer 1_2: to prevent data overfitting of the standard convolutional layer 1_1 after ReLU activation, local response normalization is performed; and
a maximum pooling layer 1_3: a pooling layer is then connected to perform maximum pooling on an image outputted after convolution, a size of the pooling layer is 3λ3, a step length is 2, and an output feature map of the first layer is obtained, which has a dimension of
Q
8
×
Q
′
8
×
96
;
2) a standard convolutional layer 2_1: an input of the convolutional layer 2 is the output feature map of the convolutional layer 1, the number of convolutional kernels is 256, the size of the convolutional kernels is: 5×5, edges are expanded with padding=2, and a step length is 1; after ReLU activation, a feature map with a dimension of
Q
8
×
Q
′
8
×
2
⁢
5
⁢
6
is output;
a local response normalization layer 2_1: to prevent data overfitting of the standard convolutional layer 2_1 after ReLU activation, local response normalization is performed; and
a maximum pooling layer 2_3: the pooling layer is then connected to perform maximum pooling on an image outputted after convolution, a size of the pooling layer is 3λ3, a step length is 2, and an output feature map of a second layer is obtained, which has a dimension of
Q
1
⁢
6
×
Q
′
1
⁢
6
×
256
;
3) a standard convolutional layer 3_1: an input of the convolutional layer 3 is the output feature map of the convolutional layer 2, the number of convolutional kernels is 384, the size of the convolutional kernels is: 3×3, edges are expanded with padding=1, a step length is 1, and after ReLU activation, an output feature map of a third layer is obtained, which has a dimension of
Q
1
⁢
6
×
Q
′
1
⁢
6
×
384
;
4) a standard convolutional layer 4_1: an input of the convolutional layer 4 is the output feature map of the convolutional layer 3, the number of convolutional kernels is 384, the size of the convolutional kernels is: 3×3, edges are expanded with padding=1, a step length is 1, and after ReLU activation, an output feature map of a fourth layer is obtained, which has a dimension of
Q
1
⁢
6
×
Q
′
1
⁢
6
×
384
;
5) a standard convolutional layer 5_1: an input of the convolutional layer 5 is the output feature map of the convolutional layer 4, the number of convolutional kernels is 256, the size of the convolutional kernels is: 3×3, edges are expanded with padding=1, and a step length is 1; after ReLU activation, a feature map with a dimension of
Q
1
⁢
6
×
Q
′
1
⁢
6
×
2
⁢
5
⁢
6
is output; and
a maximum pooling layer 5_2: a pooling layer is then connected to perform maximum pooling, a size of the pooling layer is 3×3, a step length is 2, and an output feature map of a fifth layer is obtained, which has a dimension of
Q
3
⁢
2
×
Q
′
3
⁢
2
×
256
;
6) a fully convolutional layer 6_1: to allow an input picture to be any size larger than a certain size, an input of the fully convolutional layer 6 is the output feature map of the convolutional layer 5, the number of convolutional kernels is 4096, the size of the convolutional kernels is: 1×1, edge expansion is not performed, a step length is 1, and a dimension of the output feature map is:
Q
3
⁢
2
×
Q
′
3
⁢
2
×
4
⁢
096
;
7) a fully convolutional layer 7_1: an input of the fully convolutional layer 7 is the output feature map of the convolutional layer 6, the number of convolutional kernels is 4096, the size of the convolutional kernels is: 1×1, edge expansion is not performed, a step length is 1, and a dimension of the output feature map is:
Q
3
⁢
2
×
Q
′
3
⁢
2
×
4096
;
8) upsampling and skip structure processing: 32-times upsampling is performed on the output feature map of the fully convolutional layer 7_1, and 16-times upsampling is performed on the output feature map of the standard convolutional layer 4_1, to obtain an image of the same size as the original input image; and due to the too small output feature map of the fully convolutional layer 7_1 and excessive detail losses, in order to enable an output image of the fully convolutional layer 7_1 have richer global information and more local details, a skip structure is adopted, that is, a 32-times upsampling image of the output feature map of the fully convolutional layer 7_1 and a 16-times upsampling image of the output feature map of the standard convolutional layer 4_1 are added and fused to realize global prediction of the images while predicting local image details, and a Q×Q′ segmented image is outputted as an input of the next layer;
9) an LSTM layer: in consideration of certain continuity of the pedestrian crossing behavior, in order to utilize the temporal continuity of the pedestrian behaviors, the LSTM layer is selected, an input dimension is Q, a time step is Q′, and an output dimension is 3; and
10) output: the reward function values corresponding to the three behaviors of walking, running, and stopping of the pedestrians in the status are outputted; and q(s, walking), q(s, running), q(s, stopping), S represents a current status of the pedestrians, and a predicted value q_value={q(s, walking), q(s, stopping), q(s, running)} of the behavioral reward function is composed of q(s, walking), q(s, running), q(s, stopping).

### Claim 4 (depends on 3)

The method for predicting the pedestrian crossing behavior for the intersection according to claim 3, wherein step 3: training the fully convolutional neural network-long-short term memory network (FCN-LSTM) model based on reinforcement learning, is specifically as follows:
training the FCN-LSTM model established in step 2 based on a reinforcement learning thought, selecting, in consideration of certain randomness of the behavior of the pedestrian in a case of crossing the street and in an iterative training process, pedestrian walking, stopping, and running behaviors randomly with a probability of ζ, and selecting the pedestrian behavior greedily with a probability of 1−ζ, that is, selecting a behavior corresponding to a maximum value of the behavior reward function outputted in 10) of step 2, so that the FCN-LSTM model is capable of learning characteristics that the pedestrian crossing behavior has certain purposiveness and different pedestrian individuals have certain randomness at the same time, wherein a specific training step is as follows:
sub-step 1: initializing q(s, a)=0, q(s, a) representing a motion value function value of the pedestrians, s representing the current status, a representing a current behavior, and a∈{walking, stopping, running}; and initializing the current status s;
sub-step 2: executing a one-step behavior, randomly generating a random number random of [0,1], and in a case of random <ζ, randomly selecting a pedestrian behavior, that is, randomly selecting a behavior from a pedestrian behavior motion set {walking, stopping, running}; and random ≥ζ, ζ being intended to be valued as 0,1, using a greedy policy to select the pedestrian behavior, namely, a behavior that maximizes the value q(s, a) from the pedestrian behavior motion set {walking, stopping, running};
sub-step 3: updating the status and the reward function values; and entering, after the pedestrian executes the one-step behavior, into a new status s′, and updating the reward function values according to formula (2) by utilizing the immediate reward function rt designed in step 1;
q(st,at)←q(st,at)+α(rt+γmaxaq(st+1,at)−q(st,at)) (2)
wherein q(st, at) represents a motion reward function value in the current status, t represents the time step, maxa q(st+1, a) represents a maximum motion reward function value in a next status, α represents an exploration rate, and α is intended to be taken as 0.1, rt represents an immediate reward value in the current status, γ represents a reward attenuation factor, that is, an impact of subsequent statuses on a current motion is decreased gradually, γ∈[0,1], and γ is intended to be valued as 0.95;
sub-step 4: taking updated q(st, at) in formula (2) as a true value, taking a maximum value of q_value={q(s, walking), q(s, stopping), q(s, running)} outputted by the FCN-LSTM model in step 2 as a predicted value, training the FCN-LSTM model established in step 2 based on a gradient descent method, and optimizing weight parameters of the FCN-LSTM model; and
sub-step 5: executing sub-step 2, sub-step 3, and sub-step 4 repeatedly until s terminates, that is, the pedestrian completes the crossing behavior.

### Claim 5 (depends on 3)

The method for predicting the pedestrian crossing behavior for the intersection according to claim 3, wherein step 4: predicting the pedestrian crossing behavior and performing hazard early-warning, is specifically executing step 3 repeatedly to complete a plurality of rounds of training of the FCN-LSTM model; and inputting an image of a camera deployed on a roadside of the intersection to the trained FCN-LSTM model, outputting q_value={q(s, walking), q(s, stopping), q(s, running)} by the FCN-LSTM model, taking a behavior corresponding to max {q(s, walking), q(s, stopping), q(s, running)} as the pedestrian crossing behavior at the intersection predicted by the present disclosure, and transmitting, in a case that the predicted behavior is walking or running according to the current status, an early-warning signal to the crossing pedestrian at the intersection to remind them of possible hazards.
