US 12,347,126 B2Granted

Non-contact visual monitoring system and method for a flexible protective structure against rockfall disaster

AssigneeSouthwest Jiaotong University

InventorsYongding Tian, Jiahui Tong, Yangfeng Lyu, Xiaoyu Yang, Jiaxin Chen, Lintao Ma, Hu Xu, Zhixiang Yu

Life of this patent

Estimating the term…

Loading patent…

Abstract

The invention relates to the technical field of intelligent construction and maintenance, and relates to a non-contact visual monitoring system and method for a flexible protective structure against rockfall disasters; it comprises a hardware system and a data analysis system, and the hardware system uses a multipoint distributed high-resolution high-speed camera to capture dynamic image sequences of a protective structure under the rockfall impact in a non-contact mode; the data analysis system comprises an impact deformation state full-field tracking module and a multipoint impact large deformation extraction module, and the impact deformation state full-field tracking module captures spatio-temporal changes of impact deformation of the protective system by adopting a full-field optical flow method, constructs a two-dimensional velocity amplitude distribution diagram and performs full-field spatio-temporal tracking of large deformation of the system.

Claims3

Dependent claims

A non-contact visual monitoring system for a flexible protective structure against rockfall disasters, characterized in that the non-contact visual monitoring system comprising:

a hardware system and a data analysis system, the hardware system uses a multipoint distributed high-resolution high-speed camera to collect a dynamic image sequence of a flexible protective structure under a rockfall impact in a non-contact mode;
wherein the data analysis system comprises an impact deformation state full-field tracking module and a multipoint impact large deformation extraction module for the flexible protective structure,
wherein the impact deformation state full-field tracking module for the flexible protective structure captures a full-field deformation state of the flexible protective structure under the rockfall impact by adopting a full-field optical flow method,
wherein the impact deformation state full-field tracking module
processes the dynamic image sequence of the flexible protective structure under the rockfall impact captured by the multipoint distributed high-resolution high-speed camera to obtain spatio-temporal changes of impact deformation of the flexible protective structure,
uses the full-field optical flow method to obtain a full-field velocity vector distribution of the flexible protective structure, and
builds a two-dimensional velocity amplitude distribution diagram and tracks all frame images captured by the multipoint distributed high-resolution high-speed camera, to capture a full-field and whole-process tracking of the impact deformation state of the flexible protective structure;
wherein the multipoint impact large deformation extraction module uses an image feature point detection and matching algorithm to extract a multipoint dynamic impact large deformation of the flexible protective structure under the rockfall impact and a maximum elongation based on the multipoint dynamic impact large deformation and evaluating a performance status of the flexible protective structure based on the maximum elongation.

5 more elements

A non-contact visual monitoring method for a flexible protective structure against rockfall disasters, characterized in that the non-contact visual monitoring system comprises the following steps:

1) capturing dynamic image of flexible protective structures under rockfall impact disasters;
2) tracking full-field deformation state of the flexible protective structure;
In 2), a specific method for full-field deformation state tracking of the flexible protective structure is as follows:
A grayscale of an image captured by a high-speed camera at t and t+dt is equal, namely:
I
⁢
(
x
,
y
,
t
)
=
I
⁢
(
x
+
u
⁢
(
x
,
y
)
,
y
+
v
⁢
(
x
,
y
)
,
t
+
dt
)
Where, I(x+u(x,y),y+v(x,y),t+dt) is the image grayscale information of the pixel (x,y) at t+dt I(x,y,t) is the grayscale information of the pixel (x,y) at t u(x,y) and v(x,y) are the horizontal and vertical displacement field components of the pixel (x,y) respectively:
dt is the time interval;
The grayscale information of the pixel (x,y) at t+dt is approximated by the first-order Taylor expansion formula into:
I
⁢
(
x
+
u
⁢
(
x
,
y
)
,
y
+
v
⁢
(
x
,
y
)
,
t
+
d
⁢
t
)
=
I
⁢
(
x
,
y
,
t
)
+
I
x
⁢
u
⁢
(
x
,
y
)
+
I
y
⁢
v
⁢
(
x
,
y
)
+
I
t
⁢
d
⁢
t
Where, Ix and Iy are the gray gradients of the image in the horizontal and vertical spatial domains, respectively, denoted as Ix=∂I(x,y,t)/∂x and Iy=∂I(x,y,t)/∂y:
I, is the gray gradient of the image in the temporal domain, denoted as It=∂I(x,y,t)/∂t;
obtaining displacement field vectors in both directions by using 9 pixels in the 3×3 window, namely:
{
u
⁢
(
x
,
y
)
v
⁢
(
x
,
y
)
}
=
[
∑
i
=
1
9
I
x
⁢
i
2
∑
i
=
1
9
I
x
⁢
i
⁢
I
y
⁢
i
∑
i
=
1
9
I
x
⁢
i
⁢
I
yi
∑
i
=
1
9
I
y
⁢
i
2
]
-
1
⁢
{
-
∑
i
=
1
9
I
x
⁢
i
⁢
I
t
⁢
i
-
∑
i
=
1
9
I
yi
⁢
I
t
⁢
i
}
calculating a root mean square of the velocity amplitude of the flexible protective structure at the point, namely:
V
m
⁢
a
⁢
g
(
x
,
y
)
=
(
u
⁡
(
x
,
y
)
d
⁢
t
)
2
+
(
v
⁡
(
x
,
y
)
d
⁢
t
)
2
The root mean square of the velocity amplitude of all pixels in a shooting field of view is formed into a two-dimensional matrix, and obtaining a two-dimensional velocity amplitude distribution diagram;
by comparing the two-dimensional velocity amplitude distribution diagrams in different frames, tracking a whole deformation process of the flexible protective structure under the rockfall impact;
3) extracting multipoint impact large deformation extraction of the flexible protective structure;
In 3), an extraction method of a multipoint impact large deformation of the flexible protective structure is as follows:
According to deformation characteristics of the flexible protective structure under the rockfall impact, an impact large deformation extraction area is determined, and calculating image centroids of each area separately, namely:
C
=
(
m
10
m
0
⁢
0
,
m
0
⁢
1
m
00
)
Where, m00 is the 0-order moment of the image, expressed as
m
00
=
∑
x
,
y
I
⁢
(
x
,
y
)
;
m10 and m01 are the 1-order moments of the image, respectively, expressed as
m
10
=
∑
x
,
y
xI
⁢
(
x
,
y
)
and
m
01
=
∑
x
,
y
yI
⁢
(
x
,
y
)
;
Thus, a direction angle between a detection feature point and the image centroid is calculated as:
θ
=
a
⁢
tan
⁢
2
⁢
(
m
0
⁢
1
,
m
1
⁢
0
)
A rotation matrix is formed by the direction angle, and detected point pairs are rotated to obtain feature points of the flexible protective structure with rotation invariant characteristics;
by comparing the detected feature points in different frames with the detected feature points in a reference frame, obtaining a dynamic deformation di of the flexible protective structure under the rockfall impact, and correlation coefficients of dynamic deformation curves of different feature points are defined to eliminate mismatching, namely:
R
ij
=
∑
n
=
1
N
(
d
ni
-
d
¯
í
)
⁢
(
d
n
⁢
j
-
d
¯
j
)
∑
n
=
1
N
(
d
ni
-
d
¯
i
)
2
⁢
∑
n
=
1
N
(
d
nj
-
d
j
¯
)
2
Where, dni and dnj are respectively the values of the dynamic deformation of the feature point i and the feature point j at the moment n;
di and dj are respectively the average values of the dynamic deformation of the feature point i and the feature point j;
Rij is the correlation coefficient of the feature point i and the feature point j;
A conversion coefficient is used to convert pixel displacement of the flexible protective structure under the rockfall impact into physical displacement;
according to the pinhole camera model and the principle of similar triangles, obtaining the following relationship:
s
=
f
p
×
Z
Where, f is the focal length of the high-speed camera;
Z is the actual distance between a position of the high-speed camera and a measured object;
p is the unit length of the high-speed camera;
By multiplying the conversion coefficient by the pixel displacement at multiple measuring points of the flexible protective structure under the rockfall impact, obtaining a real displacement of the flexible protective structure, and then extracting a maximum elongation based on the real displacement and evaluating a performance status-of the flexible protective structure based on the maximum elongation.

412 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/261,620Filed 5 Jun 2023
Priority: CN 202210638989.37 Jun 2022
Published: US 2024/0273744 A115 Aug 2024
Granted: 1 Jul 2025
Term ends: Not estimatedThe source records don’t support an estimate.

Classification10 codes

G06T 7/579
From multiple images: from motion
G01B 11/16
Measuring arrangements characterised by the use of optical techniques: for measuring the deformation in a solid, e.g. optical strain gauge
G06T 7/207
Analysis of motion: for motion estimation over a hierarchy of resolutions
G06T 7/269
Analysis of motion: using gradient-based methods