US 12,743,669 B2 · Method for controlling actual construction of a construction site based on multi-modality temporal information fusion, and electronic device

Abstract

An engineering progress determination method based on a multi-modality temporal information fusion includes: acquiring current engineering progress information of multiple modalities and corresponding historical engineering progress information for current engineering progress information of each modality; subsequently, extracting modality-specific temporal features from both current engineering progress information and historical engineering progress information; these modality-specific temporal features are then fed into a dimension decoupling-based multi-modality fusion module, resulting in fused multi-modality temporal information; and the current engineering progress is then determined based on this fused information.

A method for controlling actual construction of a construction site based on a multi-modality temporal information fusion, performed by at least one processor of an electronic device and comprising:

acquiring current engineering progress information and historical engineering progress information corresponding to each current engineering progress information from a local database, wherein the current engineering progress information comprises engineering progress information of a current month, and the historical engineering progress information comprises engineering progress information of previous several months of the current month;
separately determining modality-specific temporal features according to the each current engineering progress information and the historical engineering progress information corresponding to the each current engineering progress information;
inputting the modality-specific temporal features into a dimension decoupling-based multi-modality fusion module to obtain fused multi-modality temporal information;
and
expanding the fused multi-modality temporal information and determining a current engineering progress according to the fused multi-modality temporal information by using a linear predictor;
and
controlling actual construction of the construction site based on determined current engineering progress in real time, wherein the actual construction of the construction site comprises the use of material on the construction site;
wherein the current engineering progress information comprises at least two of the construction site image information, construction site personnel composition information, or construction site building material warehousing information;
and
the modality-specific temporal features comprise at least two of temporal construction site image information, temporal construction site personnel composition information, or temporal construction site building material warehousing information;
wherein separately determining the modality-specific temporal features according to the each current engineering progress information and the historical engineering progress information corresponding to the each current engineering progress information comprises:
sorting the each current engineering progress information and the historical project progress information corresponding to the each current engineering progress information according to a time sequence, extracting features of the engineering progress information of the current month and the engineering progress information of the previous several months by using a convolutional neural network to obtain a feature of the engineering progress information of the current month and features of the engineering progress information of the previous several months, and obtaining temporal engineering progress information according to a temporal data formed according to the feature of the engineering progress information of the current month and the features of the engineering progress information of the previous several months;
inputting the temporal engineering progress information into a modality-specific temporal feature extraction module, capturing, by the modality-specific temporal feature extraction module, based on a self-attention mechanism, information that satisfies a timing difference threshold in the temporal engineering progress information, and allocating a target weight to the information that satisfies the timing difference threshold;
and
obtaining the modality-specific temporal features according to the target weight;
wherein expanding the fused multi-modality temporal information and determining the current engineering progress according to the fused multi-modality temporal information by using the linear predictor comprises:
expanding the fused multi-modality temporal information into one one-dimensional vector;
and inputting the one-dimensional vector into a classifier to obtain the current engineering progress;
during a process of inputting the temporal engineering progress information into a modality-specific temporal feature extraction module for a feature extraction to obtain the modality-specific temporal features, for any input X of a modality, the modality-specific temporal feature extraction module is configured to perform the following operations:
K
=
W
K
*
LN
(
X
)
;
Q
=
W
Q
*
L
⁢
N
⁡
(
X
)
;
V
=
W
V
*
L
⁢
N
⁡
(
X
)
;
V
=
Softmax
(
K
*
Q
T
)
d
k
*
V
+
X
;
Output
=
Linear
(
Tanh
⁡
(
Linear
(
LN
(
V
)
)
)
)
+
V
;
where Wk, WQ and WV are all learned parameter matrices, dk is a number of feature dimensions of K, LN ( ) is a layer regularization operation, Linear ( ) is a linear layer with offset, and both Softmax ( ) and Tanh ( ) are corresponding activation functions;
during a process of inputting the modality-specific temporal features into a dimension decoupling-based multi-modality fusion module to obtain fused multi-modality temporal information, three inputs of X1, X2, and X3 of three modalities are operated based on the dimension decoupling-based multi-modality fusion module in following steps:
X
a
=
Concat
(
X
1
,
X
2
,
X
3
)
∈
R
M
*
D
*
L
;
X
L
=
reshape
(
X
a
,
[
M
*
D
,
L
]
)
∈
R
MD
*
L
;
X
L
″
=
Linear
(
ReLU
⁡
(
Linear
(
X
L
)
+
Bias
L
′
)
)
+
Bias
L
″
∈
R
MD
*
L
″
;
X
M
=
reshape
(
X
L
″
,
[
D
*
L
″
,
M
]
)
∈
R
DL
″
*
M
;
X
M
″
=
Linear
(
ReLU
⁡
(
Linear
(
X
M
)
+
Bias
M
′
)
)
+
Bias
M
″
∈
R
DL
″
*
M
″
;
X
D
=
reshape
(
X
M
″
,
[
L
″
*
M
″
,
D
]
)
∈
R
L
″
⁢
M
″
*
D
;
X
D
″
=
Linear
(
ReLU
⁡
(
Linear
(
X
D
)
+
Bias
D
′
)
)
+
Bias
D
″
∈
R
L
″
⁢
M
″
*
D
″
;
Output
=
reshape
(
X
D
″
,
[
M
″
,
L
″
,
D
″
]
;
where Concat ( ) is a matrix splicing function, reshape ( ) is a matrix element rearrangement function, ReLU ( ) is a corresponding activation function, Bias is a learnable induction paranoid vector, and M, L, D are modality-specific temporal features of multi-modality respectively;
wherein the inputting the modality-specific temporal features into a dimension decoupling-based multi-modality fusion module to obtain fused multi-modality temporal information comprises:
combining the modality-specific temporal features into one three-dimensional space tensor through the dimension decoupling-based multi-modality fusion module;
and
performing a dimension decoupling on the one three-dimensional space tensor, and obtaining multi-modality temporal information in a decoupled three-dimensional space tensor.

309 more elements

Method for controlling actual construction of a construction site based on multi-modality temporal information fusion, and electronic device

Life of this patent

Method for controlling actual construction of a construction site based on multi-modality temporal information fusion, and electronic device

Life of this patent

Abstract

Claims6