US 2022/0092797 A1 · Intelligent Vehicle Trajectory Measurement Method Based on Binocular Stereo Vision System

1:

A method for intelligently measuring vehicle trajectory based on a binocular stereo vision system, comprising:
step 1, inputting a dataset of images containing license plates into an SSD neural network, and training the SSD neural network with license plate as the detection feature to obtain a license plate recognition model;
step 2, installing the binocular stereo vision system on the right side, middle or above a lane, calibrating the binocular stereo vision system to acquire the internal and external parameters of two cameras;
and recording videos of moving target vehicles by the calibrated binocular stereo vision system;
step 3, detecting the license plates in the video frames recorded in step 2 with the license plate recognition model trained in step 1, and locating the license plate positions of the target vehicles;
step 4, performing feature point extraction and stereo matching on the license plates in the subsequent frames of the same camera by a feature-based matching algorithm, and reserving correct matching points after filtering with a homography matrix;
performing feature point extraction and stereo matching on the license plates in the corresponding left-view and right-view video frames of the binocular stereo vision system by a feature-based matching algorithm, and reserving correct matching points after filtering with a homography matrix;
step 5, screening the reserved matching points in step 4 by the distance measurement method of the binocular stereo vision system, and reserving the one closest to the license plate center as the position of the target vehicle in the current frame;
step 6, performing stereo measurement on the screened matching points by the binocular stereo vision system to get the spatial position coordinates of the vehicle in the video frames;
and generating the moving trajectory of the vehicle in time sequence, wherein the calibration method for the binocular stereo vision system in the step 2 comprises:
calibrating the two cameras with Zhengyou Zhang's calibration method to acquire their respective parameters such as coordinates of optical center, focal length, scale factor and/or lens distortion;
after acquiring the parameters of the two cameras, calibrating the binocular stereo vision system with Zhengyou Zhang's calibration method again to acquire the displacement and rotation angle of the right-view camera relative to the left-view camera, wherein the calibration method for the binocular stereo vision system with Zhengyou Zhang's calibration method comprises:
taking the left-view camera as the reference, calibrating with Zhengyou Zhang's calibration method to acquire the external parameters (relative translation vector T1=(l,m,n)T and relative rotation vector V=(α,β,γ)T)) of the right-view camera and the internal parameters (focal length, optical-axis angle, and distortion) of the two cameras themselves, wherein, l,m,n refer to the displacements of the right-view camera relative to the left-view camera in the three directions of x, y and z respectively, α,β,γ refer to the rotation angles of the right-view camera relative to the left-view camera around the three axes of x, y and z respectively, wherein the convergence point of the binocular stereo vision system is determined according to the external parameters:
{
B
=
l
2
+
m
2
+
n
2
ɛ
=
β
,
wherein, B is the baseline length between the two cameras, and E indicates the angle between the optical axes of the two cameras, wherein the transformation model is established for the target angle. The imaging points of the same space point in the two cameras are respectively called the left corresponding point and the right corresponding point, which are intersections of the left-view camera and the right-view camera's optical axes and their respective imaging planes. a′ represents the difference between the left corresponding point and the convergence point in the u direction of the image coordinate system, and b′ represents the difference between the right corresponding point and the convergence point in the u direction of the image coordinate system. If the left corresponding point or the right corresponding point is on the left side of the convergence point, the difference value is less than 0, otherwise the difference is greater than 0, wherein the optical axes are perpendicular to their respective imaging planes. The line connecting the optical center to the target point is called the corresponding axis. The angles a and b between the corresponding axes and the optical axes can be calculated as:
{
a
=
arctan
⁡
(
a
′
/
f
l
)
b
=
arctan
⁡
(
b
′
/
f
r
)
⁢
,
wherein, fl and fr indicate the focal length of the left-view camera and the right-view camera respectively, wherein when the target point is in region I, the target angle c can be calculated as
{
c
=
ɛ
+

b

-

a

=
ɛ
-
b
+
a
a
<
0
,
b
<
0
&
⁢
⁢
ɛ
⁢
>
0
;
Assuming the world coordinates of the target point P in region I are (x, y, z), and a depth calculation model for the target point P are capable of being established as:
tan
⁢
⁢
a
=
x
z
;
tan
⁡
(
c
-
a
)
=
l
+
x
z
-
n
,
a
<
0
,
⁢
then
z
=
n
⁢
⁢
tan
⁡
(
ɛ
-
b
)
+
l
tan
⁡
(
ɛ
-
b
)
+
tan
⁢
⁢
a
,
wherein the world coordinate x can be calculated with the depth calculation model for the target point P as:
x
=
n
⁢
⁢
tan
⁡
(
ɛ
-
b
)
+
l
tan
⁡
(
ɛ
-
b
)
+
tan
⁢
a
⁢
tan
⁢
a
,
wherein the world coordinate y can be calculated according to the relation between the left corresponding point and the right corresponding point in the image coordinate system and the relation between the image coordinate system and the world coordinate system, wherein the world coordinates of the target point P are:
{
x
=
n
⁢
tan
⁡
(
ɛ
-
b
)
+
l
tan
⁡
(
ɛ
-
b
)
+
tan
⁢
a
⁢
tan
⁢
a
y
=
n
⁢
tan
⁡
(
ɛ
-
b
)
+
l
tan
⁡
(
ɛ
-
b
)
+
tan
⁢
a
·
v
′
f
l
z
=
n
⁢
tan
⁡
(
ɛ
-
b
)
+
l
tan
⁡
(
ɛ
-
b
)
+
tan
⁢
a
;
wherein, v′ indicates the pixel difference between the target point and the image center in the longitudinal direction of the image coordinate system, and fl is the focal length of the left-view camera, similarly, the world coordinates of the target point in region II, region III and region IV are capable of calculated.

258 more elements

5:

The method for intelligently measuring vehicle trajectory based on a binocular stereo vision system according to claim 1, wherein the feature-based matching algorithm is a SURF feature extracting and matching algorithm. The local features of the video frames are described by the SURF descriptors, wherein the homography matrix describes the relationship between two images of the same object taken in the different views. Assuming that the relationship of the two images is perspective transformation, then the homography matrix H is:
H
=
[
h
1
⁢
1
h
1
⁢
2
h
1
⁢
3
h
2
⁢
1
h
2
⁢
2
h
2
⁢
3
h
31
h
3
⁢
2
1
]
;
then
⁢
[
x
/
y
/
1
]
=
[
h
1
⁢
1
h
1
⁢
2
h
1
⁢
3
h
2
⁢
1
h
2
⁢
2
h
2
⁢
3
h
31
h
3
⁢
2
1
]
⁡
[
x
y
1
]
,
wherein x′, y′, 1 and x, y, 1 represent the coordinates of the two corresponding points before and after the perspective transformation respectively, and h11-32 are transformation parameters to be calculated to calculate the eight transformation parameters h11-32 in the homography matrix H, at least four pairs of matching points are needed. The process is as follows:
[
x
1
,
y
1
,
1
,
0
,
0
,
0
,
-
x
1
′
⁢
x
1
,
-
x
1
′
⁢
y
1
0
,
0
,
0
,
x
1
,
y
1
,
1
,
-
y
1
′
⁢
x
1
,
-
y
1
′
⁢
y
1
x
2
,
y
2
,
1
,
0
,
0
,
0
,
-
x
2
′
⁢
x
2
,
-
x
2
′
⁢
y
2
0
,
0
,
0
,
x
2
,
y
2
,
1
,
-
y
2
′
⁢
x
2
,
-
y
2
′
⁢
y
2
x
3
,
y
3
,
1
,
0
,
0
,
0
,
-
x
3
′
⁢
x
3
,
-
x
3
′
⁢
y
3
0
,
0
,
0
,
x
3
,
y
3
,
1
,
-
y
3
′
⁢
x
3
,
-
y
3
′
⁢
y
3
x
4
,
y
4
,
1
,
0
,
0
,
0
,
-
x
4
′
⁢
x
4
,
-
x
4
′
⁢
y
4
0
,
0
,
0
,
x
4
,
y
4
,
1
,
-
y
4
′
⁢
x
4
,
-
y
4
′
⁢
y
4
]
⁡
[
h
11
h
12
h
13
h
21
h
22
h
23
h
31
h
32
]
=
[
x
1
′
y
1
′
x
2
′
y
2
′
x
3
′
y
3
′
x
4
′
y
4
′
]
;
each time, four pairs of matching points are selected from all the matching points to calculate the homography matrix H;
then the homography matrix with the maximum number of accurate matching points is selected as the most accurate homography matrix H to check the accuracy of the matrix H, the Euclidean distance between the corresponding matching points is calculated:

[
x
i
⁢
1
′
y
i
⁢
1
′
1
]
-
H
⁡
[
x
i
⁢
1
y
i
⁢
1
1
]

≤
t
wherein xi1′, yi1′, 1 and xi1, yi1, 1 are the coordinates of the matching points before and after the perspective transformation, t is the Euclidean distance threshold, and i1=1,2,3,4 wherein the smaller the Euclidean distance, the higher the matching accuracy of the two corresponding matching points.

400 more elements

Intelligent Vehicle Trajectory Measurement Method Based on Binocular Stereo Vision System

Life of this patent

Intelligent Vehicle Trajectory Measurement Method Based on Binocular Stereo Vision System

Life of this patent

Abstract

Claims12