All-weather target detection method based on vision and millimeter wave fusion

An all-weather target detection method based on a vision and millimeter wave fusion, comprising:

• simultaneously acquiring continuous image data and point cloud data by using two types of sensors comprising a vehicle-mounted camera and a millimeter wave radar;
• pre-processing the image data and the point cloud data;
• fusing pre-processed image data and pre-processed point cloud data by using a pre-established fusion model, and outputting a fused feature map;
• and
• inputting the fused feature map into a YOLOv5 detection network for a detection, and outputting a target detection result by a non-maximum suppression;
• wherein the pre-established fusion model comprises a feature extraction module and a feature fusion module;
• the feature extraction module comprises a first feature extraction channel, a second feature extraction channel, and three feature stitching units;
• the first feature extraction channel extracts features of scattering cross-section intensity information of a radar point cloud by using three sequentially connected convolution units;
• and the second feature extraction channel extracts features of the pre-processed image data by using a CSPDarkNet53 comprising three convolution units;
• a first convolution unit of the first feature extraction channel processes the scattering cross-section intensity information of the pre-processed point cloud data to obtain a first radar cross section (RCS) feature, and outputs the first RCS feature to a first feature stitching unit;
• a first convolution unit of the CSPDarkNet53 processes the pre-processed image data to obtain a first visual feature, and outputs the first visual feature to the first feature stitching unit;
• the first feature stitching unit stitches the first RCS feature and the first visual feature to generate a first fused feature map and output the first fused feature map to the feature fusion module;
• a second convolution unit of the first feature extraction channel processes the first RCS feature to obtain a second RCS feature, and outputs the second RCS feature to a second feature stitching unit;
• a second convolution unit of the CSPDarkNet53 processes the first visual feature to obtain a second visual feature, and outputs the second visual feature to the second feature stitching unit;
• the second feature stitching unit stitches the second RCS feature and the second visual feature to generate a second fused feature map and output the second fused feature map to the feature fusion module;
• a third convolution unit of the first feature extraction channel processes the second RCS feature to obtain a third RCS feature, and outputs the third RCS feature to a third feature stitching unit;
• a third convolution unit of the CSPDarkNet53 processes the second visual feature to obtain a third visual feature, and outputs the third visual feature to a third feature stitching unit;
• the third feature stitching unit processes the third RCS feature and the third visual feature to generate a third fused feature map and output the third fused feature map to the feature fusion module;
• the feature fusion module comprises a third feature extraction channel, three fusion units, and a PANet structure;
• the third feature extraction channel configures three sequentially connected convolution units;
• a first convolution unit of the third feature extraction channel processes distance information of the pre-processed point cloud data and velocity information of the pre-processed point cloud data to obtain a first point cloud feature, and outputs the first point cloud feature to a first fusion unit;
• the first fusion unit performs a pixel-level superimposition on the first point cloud feature and the first fused feature map to generate a fourth fused feature map and output the fourth fused feature map to the PANet structure;
• a second convolution unit of the third feature extraction channel processes the first point cloud feature to obtain a second point cloud feature, and outputs the second point cloud feature to a second fusion unit;
• the second fusion unit performs the pixel-level superimposition on the second point cloud feature and the second fused feature map to generate a fifth fused feature map and output the fifth fused feature map to the PANet structure;
• a third convolution unit of the third feature extraction channel processes the second point cloud feature to obtain a third point cloud feature, and outputs the third point cloud feature to a third fusion unit;
• the third fusion unit performs the pixel-level superimposition on the third point cloud feature and the third fused feature map to generate a sixth fused feature map and output the sixth fused feature map to the PANet structure;
• and
• the fourth fused feature map, the fifth fused feature map and the sixth fused feature map pass through the PANet structure to output a first feature map, a second feature map and a third feature map respectively.