US 2024/0176933 A1 · DIGITAL TWIN BASED METHOD FOR MONITORING BEHAVIOR OF PASSENGER ON ESCALATOR

Abstract

A digital twin based method for monitoring behavior of a passenger on an escalator includes: first, constructing a digital twin virtual scene of the escalator carrying the passenger; constructing a virtual person model in a physics engine, generating various types of risky behavior in reality, and outputting different types of behavior of a virtual person; second, through a human posture recognition method for monitoring behavior of a passenger, extracting a feature map as an input by means of a visual geometry group 19 (VGG19) pre-training network, to enter part affinity fields (PAFs) and a part confidence map (PCM), so as to complete recognition of human postures; classifying the human postures with a support vector machine; and performing emergency stop control on the escalator under risky behavior of the passenger according to a result of posture classification.

A digital twin based method for monitoring behavior of a passenger on an escalator, comprising:

step 1:
constructing a digital twin virtual scene of the escalator for carrying the passenger, comprising:
step (1.1) drawing a geometric model of the escalator and the passenger:
drawing the geometric model by a three-dimensional modeling software, wherein the digital twin virtual scene determines a final realization effect of the digital twin scene and a degree of fidelity of the virtual scene based on the geometric model;
step (1.2) constructing the digital twin virtual scene:
constructing a subject same with a real subject in reality for a complex step system in the geometric model, defining an attribute of a motion component, and defining a data interface of the motion component;
and
step (1.3) driving the geometric model;
step 2:
constructing a virtual person model in a physics engine, generating a virtual person model generating different actions corresponding to various types of risky behavior in the reality, making a virtual camera, and outputting different types of behavior of a virtual person, wherein the step 2 comprises:
step (2.1) obtaining behavior data of an actual person, and performing a posture decomposing process to obtain 18 key points;
step (2.2) designing action simulation behavior of the virtual person, reorienting a skeleton model of the virtual person in the physics engine, wherein the reorienting the skeleton model of the virtual person in the physics engine comprises:
achieving simulation of the actual person by means of rotation and displacement in the skeleton model, matching a relation between skeleton points anew by means of matching set, and editing a space position of each key point of the virtual person, to simulate the behavior of the passenger in a real world;
and
step (2.3) adding the virtual camera in the physics engine, binding visual angle selection of the camera with the virtual person, setting a photographing parameter of the virtual camera, selecting an output save path, and using a video image of the virtual person as input of behavior posture recognition and classification;
step 3, using a visual geometry group 19 (VGG19) pre-training network by a human posture recognition method for monitoring behavior of a passenger, extracting a feature map as input and entering two branches of part point affinity fields (PAFs) and point confidence maps (PCM), and using the PAFs to express a trend of a pixel point on a posture, wherein the step 3 comprises:
step (3.1) obtaining the feature map from an original image by the VGG19 pre-training network;
step (3.2) outputting the obtained feature map to a next layer, entering the two branches, and outputting one Loss in each stage, wherein the two branches comprises a branch PAFs and a branch PCM;
step (3.3) generating and computing a confidence map S by using an image of two-dimensional key points, wherein all confidence maps S*j,k(p) are generated by k persons, Xj,k represents a jth key point of a kth person in the image, and a maximum value of the all confidence maps S*j,k(p) is a finally obtained confidence map of key point parts of a plurality of persons;
wherein a predicted value at p is:
S
j
,
k
*
(
p
)
=
exp
⁢
(
-

p
-
X
j
,
k

2
2
δ
2
)
wherein exp is a natural constant e, & is a diffusion of a control peak, and ∥p−Xj,k∥22 is a square of a vector modulus value from the point p to a position j of the kth person;
step (3.4) defining that Xi,k and Xj,k represent two key points, and the pixel point p step (3.4) defining that Xix and is on a limb, such that a value of L*c,k(p) is a unit vector from i to j of the kth person, wherein a vector field of the point p is:
L
c
,
k
*
(
p
)
=
{
(
X
j
,
k
-
X
i
,
k
)

X
j
,
k
-
X
i
,
k

2
,
p
∈
L
c
,
k
*
⁢
(
p
)
0
,
others
wherein ∥Xj,k−Xi,k∥2 is a limb length from a position j to position i of the kth person;
step (3.5) obtaining an average affinity field of all the persons as:
L
c
*
(
p
)
=
1
n
c
(
p
)
⁢
L
c
,
k
*
(
p
)
wherein nc(p) represents a number of non-zero vectors at p in all the persons;
step (3.6) computing a score of a limb by means of the following formula in a multi-person scene, and searching for a situation with a maximum association confidence coefficient;
E
=
∫
u
=
0
u
=
1
L
c
(
p
⁡
(
u
)
)
⁢
d
j
⁢
2
-
d
j
⁢
1

d
j
⁢
2
-
d
j
⁢
1

2
⁢
du
wherein E is the association confidence coefficient, ∥dj2−dj2∥2 is a distance between body parts dj2, dj1, and p(u) interpolates positions of the body parts dj2, dj1:
p(u)=(1−u)dj1+dj2
wherein an integral value is approximated by means of sampling and equally spaced sums over u;
and
step (3.7) changing a multi-person detection into a bipartite graph matching to obtain an optimal solution of connected points, obtaining all limb prediction results, and connecting all key points of a human body;
wherein the bipartite graph matching is a subset of edges selected in such a way that two edges share no node to find a matched maximum weight for a selected edge;
step 4, performing posture classification based on a support vector machine (SVM), using 1-V-1(one-versus-one) to classify a plurality of human postures, wherein designing one SVM classifier for every two classes, and k(k−1)/2 classifiers are needed in total;
and
step 5:
performing emergency stop control on the escalator under risky behavior.

171 more elements

Digital twin based method for monitoring behavior of passenger on escalator

Life of this patent

Digital twin based method for monitoring behavior of passenger on escalator

Life of this patent

Abstract

Claims8