US 11,900,619 B2 · Intelligent vehicle trajectory measurement method based on binocular stereo vision system

Abstract

The invention provides a method for intelligently measuring vehicle trajectory based on a binocular stereo vision system, including the following steps: inputting a dataset into an SSD (Single Shot Multibox Detector) neural network to train a license plate recognition model; calibrating the binocular stereo vision system, and recording videos of moving target vehicles; detecting the license plates in the video frames with the license plate recognition model; performing stereo matching on the license plates in the subsequent frames of the same camera and in the corresponding left-view and right-view video frames by a feature-based matching algorithm; reserving correct matching point pairs after filtering with a homography matrix; screening the reserved matching point pairs, and reserving the one closest to the license plate center as the position of the target vehicle in the current frame; performing stereo measurement on the screened and reserved matching point pairs to get the spatial position coordinates of the vehicle in the video frames; and generating the moving trajectory of the vehicle in time sequence. The present invention is easy to install and adjust, and can simultaneously measure multiple target vehicles in multiple directions and on multiple lanes.

A method for intelligently measuring vehicle trajectory based on a binocular stereo vision system, comprising:

step 1, inputting a dataset of images containing license plates into a SSD neural network, and training the SSD neural network with license plate as a detection feature to obtain a license plate recognition model;
step 2, installing the binocular stereo vision system on the right side, middle or above a lane, calibrating the binocular stereo vision system to acquire internal and external parameters of two cameras;
and recording videos of moving target vehicles by the calibrated binocular stereo vision system;
step 3, detecting license plates in video frames recorded in step 2 with the license plate recognition model trained in step 1, and locating license plate positions of target vehicles;
step 4, performing feature point extraction and stereo matching on license plates in subsequent frames of a same camera by a feature-based matching algorithm, and reserving correct matching points after filtering with a homography matrix;
performing feature point extraction and stereo matching on license plates in corresponding left-view and right-view video frames of the binocular stereo vision system by a feature-based matching algorithm, and reserving correct matching points after filtering with a homography matrix;
step 5, screening the reserved matching points in step 4 by a distance measurement method of the binocular stereo vision system, and reserving one closest to a license plate center as the position of a target vehicle in a current frame;
step 6, performing stereo measurement on the screened matching points by the binocular stereo vision system to get spatial position coordinates of vehicle in the video frames;
and generating a moving trajectory of the vehicle in time sequence, wherein a calibration method for the binocular stereo vision system in the step 2 comprises:
calibrating the two cameras with Zhengyou Zhang's calibration method to acquire their respective parameters such as coordinates of optical center, focal length, scale factor and/or lens distortion;
after acquiring the parameters of the two cameras, calibrating the binocular stereo vision system with Zhengyou Zhang's calibration method again to acquire a displacement and rotation angle of a right-view camera relative to a left-view camera, wherein the calibration method for the binocular stereo vision system with Zhengyou Zhang's calibration method comprises:
taking the left-view camera as the reference, calibrating with Zhengyou Zhang's calibration method to acquire the external parameters (relative translation vector T1=(l,m,n)T and relative rotation vector V=(α,β,γ)T)) of the right-view camera and the internal parameters (focal length, optical-axis angle, and distortion) of the two cameras themselves, wherein, l,m,n refer to displacements of the right-view camera relative to the left-view camera in three directions of x, y and z respectively, α,β,γ refer to the rotation angles of the right-view camera relative to the left-view camera around three axes of x, y and z respectively, wherein a convergence point of the binocular stereo vision system is determined according to the external parameters:
{
B
=
l
2
+
m
2
+
n
2
ε
=
β
,
wherein, B is a baseline length between the two cameras, and ε indicates the angle between the optical axes of the two cameras, wherein a transformation model is established for a target angle, and imaging points of the same space point in the two cameras are respectively called a left corresponding point and a right corresponding point, which are intersections of the left-view camera and the right-view camera's optical axes and their respective imaging planes, α′ represents a difference between the left corresponding point and the convergence point in the u direction of the image coordinate system, and b′ represents a difference between the right corresponding point and the convergence point in the u direction of the image coordinate system when the left corresponding point or the right corresponding point is on the left side of the convergence point, a difference value is less than 0, otherwise a difference is greater than 0, wherein the optical axes are perpendicular to their respective imaging planes, a line connecting the optical center to the target point is called corresponding axis, and angles α and b between the corresponding axes and the optical axes is calculated as:
{
a
=
arctan
⁡
(
a
′
/
f
l
)
b
=
arctan
⁡
(
b
′
/
f
r
)
,
wherein, fl and fr indicate a focal length of the left-view camera and the right-view camera respectively, wherein when the target point is in region I, a target angle c can be calculated as
{
c
=
ε
+
❘
"\[LeftBracketingBar]"
b
❘
"\[RightBracketingBar]"
-
❘
"\[LeftBracketingBar]"
a
❘
"\[RightBracketingBar]"
=
ε
-
b
+
a
a
<
0
,
b
<
0
&
⁢
ε
>
0
;
the world coordinates of a target point P in region I are (x, y, z), and a depth calculation model for the target point P are being established as:
tan
⁢
a
=
x
z
;
⁢
tan
⁡
(
c
-
a
)
=
l
+
x
z
-
n
,
a
<
0
,
then
⁢
z
=
n
⁢
tan
⁡
(
ε
-
b
)
+
l
tan
⁡
(
ε
-
b
)
+
tan
⁢
a
,
wherein a world coordinate x can be calculated with the depth calculation model for the target point P as:
x
=
n
⁢
tan
⁡
(
ε
-
b
)
+
l
tan
⁡
(
ε
-
b
)
+
tan
⁢
a
⁢
tan
⁢
a
,
wherein a world coordinate y can be calculated according to a relation between the left corresponding point and the right corresponding point in the image coordinate system and a relation between the image coordinate system and a world coordinate system, wherein the world coordinates of the target point P are:
{
x
=
n
⁢
tan
⁡
(
ε
-
b
)
+
l
tan
⁡
(
ε
-
b
)
+
tan
⁢
a
⁢
tan
⁢
a
y
=
n
⁢
tan
⁡
(
ε
-
b
)
+
l
tan
⁡
(
ε
-
b
)
+
tan
⁢
a
·
v
′
f
l
z
=
n
⁢
tan
⁡
(
ε
-
b
)
+
l
tan
⁡
(
ε
-
b
)
+
tan
⁢
a
;
wherein, v′ indicates the pixel difference between the target point and the image center in a longitudinal direction of the image coordinate system, and fl is the focal length of the left-view camera, similarly, world coordinates of the target point in region II, region III and region IV are calculated.

255 more elements

Intelligent vehicle trajectory measurement method based on binocular stereo vision system

Life of this patent

Intelligent vehicle trajectory measurement method based on binocular stereo vision system

Life of this patent

Abstract

Claims12