# US 11,908,222 B1 · Occluded pedestrian re-identification method based on pose estimation and background suppression

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,222 B1
- Document ID: US11908222B1
- Status: Granted
- Kind: B1
- Application number: 18/488,146
- Filed: 2023-10-17
- Granted: 2024-02-20
- Earliest claimed: 2022-12-13
- Assignees: Hangzhou Dianzi University
- Applicants: Hangzhou Dianzi University
- Inventors: Ming Jiang; Lingjie He; Min Zhang
- Web page: https://patentgrep.com/patent/US11908222B1

## CPC

- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06T 5/20: using local operators
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/462: {Salient features, e.g. scale invariant feature transforms [SIFT]}
- G06V 10/82: using neural networks
- Y02T 10/40: Engine management systems
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- Y02T: CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION

## Abstract

The present application relates to an occluded pedestrian re-identification method, including steps of obtaining global features and local features of occluded pedestrians, and recombining the local features into a local feature map; obtaining a heat map of key-points of pedestrian images and a group of key-point confidences, obtaining a group of features of the pedestrian key-points by using the local feature map and the heat map; obtaining a local feature group by using the global features to enhance each key-point feature in the group of features of pedestrian key-points according to Conv, and an adjacency matrix of key-points is obtained through the key-points, the local feature group and the adjacency matrix of key-points are used as the input of GCN to obtain the final features of pedestrian key-points.

## Claims

### Claim 1

An occluded pedestrian re-identification method based on pose estimation and background suppression, comprising steps of:
step (1): constructing a pedestrian feature extraction backbone network based on ViT to obtain global features ƒcls and local features ƒƒ_local of occluded pedestrians, and recombining the local features ƒƒ_local into a local feature map ƒlocal;
step (2): obtaining a heat map of key-points of a pedestrian images ƒpos and a group of key-point confidences Vkc according to a pre-trained ViTPose, and then obtaining a group of features of pedestrian key-points ƒkeypoints by using the local feature map ƒlocal and the heat map ƒpos;
step (3): constructing a graph convolutional module of local feature enhancement composed of Conv and GCN, and then obtaining a local feature group ƒkp_en by using the global features ƒcls to enhance each key-point feature in the group of features of pedestrian key-points ƒkeypoints according to through Conv, and obtaining an adjacency matrix of key-points A through the key-points, and finally the local feature group ƒkp_en and the adjacency matrix of key-points A are used as input of GCN to obtain final features of pedestrian key-points ƒƒ_keypoints;
step (4): constructing an attention background suppression module composed of global average pooling and convolutional networks, inputting the local feature map ƒlocal obtained by step (1) and the heat map ƒpos obtained by step (2) into the attention background suppression module to output pedestrian features ƒatt_local, segmenting the pedestrian features as final features final features;
step (5): constructing an occluded pedestrian re-identification model by the pedestrian feature extraction backbone network, the ViTPose, the graph convolutional module of local feature enhancement and the attention background suppression module, and then training the occluded pedestrian re-identification module by using the global pedestrian features ƒcls, the features of pedestrian key-points ƒƒ_keypoints and the pedestrian features ƒatt_local to obtain a final occluded pedestrian re-identification model.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the pedestrian feature extraction backbone network is obtained based on an initial network of pedestrian feature extraction which adopting the ViT pre-trained on ImageNet, wherein before the images are input into ViT, extracting features from the images by a convolutional network, as shown in equation(1):
x=Conv(X) (1)
wherein X represents the pedestrian image, Conv represents the convolutional network, and x is the pedestrian features output by the convolutional network.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the step of obtaining the global features ƒcls and the local features ƒƒ_local of occluded pedestrians comprises: generating a feature map sequence xp={xpi|i=1,2, . . . N} by segmenting the obtained the pedestrian features x according to a preset patch size p, wherein N is a separable quantity, and then adding camera perspective information [CAM_VIEW] to xp, the dimension of camera perspective information is the same as xp, as shown in equation(2):
xp=xp+λ*Ecam_view (2)
wherein λ is a hyperparameter representing the weight of the camera perspective information, and Ecam_view is [CAM_VIEW] representing the camera perspective information;
adding [CLS_TOKEN] representing the global feature and the position information coding [POS_TOKEN] to xp, and then feature Z can be obtained after linear coding, as shown in equation (3):
Z=[xcls;linear(xpi)]+Epos (3)
wherein xcls is the global feature vector [CLS_TOKEN]; linear(·) is a linear coding function; Epos is [POS_TOKEN] representing the spatial position;
inputting the feature Z into the ViT, wherein the ViT is stacked by multiple Transformer blocks, as shown in equation (4) and equation (5):
Zl′=MSA(LN(Zl−1))+Z1−1l=1 . . . L (4)
Zl=MLP(LN(Zl′))+Zl′l=1 . . . L (5)
wherein Zl represents the features of the output of the Transformer block in the 1 layer, Zl′ is the intermediate result in the Transformer block, and L is the total number of layers;
MSA(·) is the multi-head attention, LN(·) is the layer normalization, and MLP(·) is the multilayer perceptron;
the network output result is the output feature of the last layer, which are the global features ƒcls ∈ R1×c and the local feature group ƒƒ_local respectively.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the acquisition of the local feature map ƒlocal comprises: rearranging the local feature group ƒƒ_local as follows:
ƒlocal reshape(ƒƒ_local) (6)
wherein reshape(·) is the rearrangement function.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the specific implementation process of step(2) comprises steps of step 2-1 to step 2-2:
step 2-1: extracting the key-points of the pedestrian images by ViTPose which is pre-trained on the CoCo dataset, and the heat map of the pedestrian key-points ƒpos and key-point set Vkc={V1, V2, . . . , VS} in the pedestrian images are obtained by ViTPose, among which, VS is the key-point of the human body obtained by the pedestrian key-point algorithm, as shown in equation(7):
ƒpos,Vkc=ViTPose(Image) (7)
wherein VS={kx, ky, kc},kx, ky are the coordinates of key-points respectively, and kc is the key-point confidence; ƒpos is the heat map of key-points output by ViTPose
step 2-2: obtaining S local features of pedestrian key-points by using the local feature map ƒlocal and the heat map ƒpos according to vector outer producting and global average pooling, as shown in equation (8):
ƒkeypoints=GAP(ƒlocal⊗ƒpos) (8)
wherein GAP is the global average pooling; the group of features of pedestrian key-points ƒkeypoints ∈ RS×C S is the number of key-points and C is the number of feature channels.

### Claim 6 (depends on 5)

The method according to claim 5, wherein obtaining the local feature group ƒkp_en in step (3) comprises:
firstly, the group of features of pedestrian key-points can be expressed as equation (9):
ƒkeypoints={ƒkeypointsi|i=0,1, . . . ,S} (9)
secondly, apply 1*1 convolution to each key-point feature and the global feature ƒcls, as shown in equation (10) and equation (11):
ƒkp_conv=Conv1×1(ƒkeypoints) (10)
ƒcls_conv=Conv1×1(ƒcls) (11)
wherein ƒkp_conv is the feature after convolution of each local feature, and ƒcls_conv is the feature after convolution of global features;
lastly, by using the obtained group of features of pedestrian key-points ƒkeypoints and global features ƒcls, the local feature group of enhanced key-points ƒkp_en is calculated by vector quantity product, softmax and addition, as shown in equation (12) and equation (13):
Vsim=Softmax(ƒkp_conv⊙ƒcls_conv) (12)
ƒkp_en=Conv(ƒkeypoints+w*(ƒcls_conv+Vsim*ƒcls)) (13)
wherein Cony is the convolution operation; w is the learnable weight; Vsim is the similarity.

### Claim 7 (depends on 6)

The method according to claim 6, wherein the method of obtaining the final features of pedestrian key-points ƒƒ_keypoints in step (3) comprises:
by using the adjacency matrix of pedestrian key-points A and the local feature group ƒkp_en as the input of the graph convolutional network, outputting the final features of pedestrian key-points ƒƒ_keypoints by the graph convolutional network, as shown in equation (14):
ƒƒ_keypoints=GCN(A,ƒkp_en) (14)
wherein GCN is the graph convolutional network, and λ is a predefined adjacency matrix of human key-points.

### Claim 8 (depends on 7)

The method according to claim 7, wherein the specific implementation process of step (4) comprises steps of step 4-1 to step 4-4:
step 4-1: performing global average pooling to the heat map of the key-points ƒpos to obtain the features of pedestrian key-points, and supplementing the features of pedestrian key-points to the local features ƒƒ_local;
step 4-2: using the local feature map ƒlocal as a graph structure, wherein the graph comprises H*W nodes and each node is a C-dimensional feature, inputting the local feature map ƒlocal into two 1*1 convolutional networks, and then transposing the output of one of the convolutional networks to construct the relationship between nodes, as shown in equation (15):
Ri,j=Conv(ƒlocal)TConv(ƒlocal) (15)
wherein Ri,j is the matrix of relational feature, and Conv is a convolutional network;
step 4-3: obtaining the features of spatial perception fs, of the corresponding relationship by using the matrix of relationship feature Ri,j, and then embedding the local feature map ƒlocal, the features of pedestrian key-points ƒpos and the features of spatial perception fs, into a link, as shown in equation (16) and equation (17):
ƒsp=Conv(Ri,j) (16)
ƒconcat=Concat[Conv(ƒlocal),Conv(ƒsp), Conv(ƒpos)] (17) wherein ƒsp are the features of spatial perception, Concat(·) is the channel link function, and ƒconcat are the connected feature vectors;
inputting the ƒconcat into a 1*1 convolutional network and Sigmod to obtain a spatial attention map fatten, and finally a final pedestrian feature map ƒatt_local is obtained by multiplying the spatial attention map fatten with the local feature map ƒlocal;
step 4-4: constructing multiple classification heads according to the pedestrian structure, and dividing the pedestrian feature map ƒatt_local into four local features, ƒ1, ƒ2, ƒ3, ƒ4, to classify the pedestrian images.

### Claim 9 (depends on 8)

The method according to claim 8, wherein the specific implementation process of step (5) comprises steps of step 5-1 to step 5-4:
step 5-1: using labeled data in the pedestrian re-identification dataset as supervision information, and using ID loss and difficult triplet loss to train the network for each training batch as shown in equation (18), wherein the ID loss uses cross entropy loss:
L
i
⁢
d
=
∑
i
=
1
N
-
q
i
⁢
log
⁡
(
p
i
)
(
18
)
wherein N is the number of pedestrian categories, qi is the supervised label and pi is the predictive label;
the difficult triplet loss randomly samples P identities, each identity extracts K instances to form a mini batch with a size of P*K; each picture xa in the batch is selected as the anchor point in turn, and the farthest positive sample picture xp and the nearest negative sample picture xn in the batch are selected to form a triple to train the network in order to enhance the generalization ability of the network, as shown in equation(19):
L
triplet
=
∑
i
=
1
P
⁢
∑
a
=
1
K
[
m
+
max
1
≤
p
≤
K
||
f
⁢
(
x
i
,
a
)
-
f
⁢
(
x
i
,
p
)
||
2
︷
hardest
⁢
positive
-
min
n
=
1
⁢
…
⁢
K
j
=
1
⁢
…
⁢
P
⁢
j
≠
i
||
f
⁡
(
x
i
,
a
)
-
f
⁡
(
x
j
,
n
)
||
2
︸
hardest
⁢
negative
]
+
(
19
)
step 5-2: training the occluded pedestrian re-identification model according to ƒ1, ƒ2, ƒ3, ƒ4 generated by the global features ƒcls, the final group of features of pedestrian key-points ƒƒ_keypoints and the pedestrian features ƒatt_local division to obtain a final model, and the specific equation can be expressed as follows:
Loss
=
L
id
(
f
cls
)
+
L
triplet
(
f
cls
)
+
1
S
⁢
∑
i
=
1
S
k
c
i
[
L
id
(
f
f_keypoints
i
)
+
L
triplet
(
f
f_keypoints
i
)
]
+
1
k
⁢
∑
i
=
1
k
⁢
(
L
id
(
f
i
)
+
L
triplet
(
f
i
)
)
(
20
)
wherein S is the number of pedestrian key-points, and kc is the key-point confidence obtained in step 2-1;
step 5-3: in the test stage, inputting images to be queried q and test set images t into the final model for feature extraction, to obtain key-point features of the images to be queried q represented by ƒq and key-point features the test set images t represented by ƒt, and using graph matching optimization comparison, as shown in equation (21):
GM
=
1
S
⁢
∑
i
=
1
S
⁢
k
c_q
i
⁢
k
c_t
i
⁢
cosine
(
f
q
i
,
f
t
i
)
(
21
)
wherein kc_qi and kc_ti are the i-th key-point confidences of the image q and t respectively; cosine is the cosine distance;
step 5-4: comparing the query image features with the test set image features, judging whether the images belong to the same class, and output the same class of pedestrian images.