Saliency Map Enhancement-Based Infrared and Visible Light Fusion Method

A saliency map enhancement-based infrared and visible light fusion method, wherein the method comprises the following steps:

• 1) obtaining registered infrared and visible light images, and respectively calibrating each lens and jointly calibrating the respective systems of the visible light binocular camera and the infrared binocular camera;
• 1-1) respectively calibrating the infrared camera and the visible light camera by the Zhangzhengyou calibration method to obtain internal parameters including focal length and principal point position and external parameters including rotation and translation of each camera;
• 1-2) calculating the positional relationship of the same plane in the visible light image and the infrared image by using the pose relationship RT of the visible light camera and the infrared camera obtained by joint calibration of the cameras and the detected checker corners, and registering the visible light image to the infrared image by using homography transformation;
• 2) converting the color space of the visible light image from an RGB image to an HSV image, extracting the value information of the color image as the input of image fusion, and retaining the original hue and saturation;
• 3) carrying out mutual guided filtering decomposition on the input infrared image and the visible light image with the color space converted, and decomposing the images into a background layer and a detail layer, wherein the background layer includes the structural information of the images, and the detail layer includes the gradient and texture information of the images;
• B=M(I,V), D=(I,V)−B
• wherein B represents the background layer, D represents the detail layer, M represents the mutual guided filtering, and I represents the infrared image;
• 4) fusing the background layer B by the saliency map-based method, subtracting each pixel from all the pixels, and taking and adding absolute values, wherein the formula is as follows:
• S(p)=|I(p)−I1|+|I(p)−I2|+|I(p)−I3|+ ⋅ ⋅ ⋅ +|I(p)−IN|
• that is
• S
• ⁡
• (
• p
• )
• =
• ∑
• i
• =
• 0
• 2
• ⁢
• 5
• ⁢
• 5
• ⁢
• M
• ⁡
• (
• i
• )
• ⁢
• 
• I
• ⁡
• (
• p
• )
• -
• i
• 
• wherein S(p) represents the salience value of the pixel, N represents the number of pixels in the image, M represents the histogram statistical formula, and I(p) represents the value of the pixel position;
• according to the obtained saliency value, obtaining the saliency map weight based on background layer fusion:
• W
• =
• S
• ⁡
• (
• p
• )
• ∑
• S
• ⁡
• (
• p
• )
• j
• wherein W represents the weight, and Sj represents the corresponding pixel value;
• then performing linearly weighted fusion of the decomposed infrared image and visible light image based on the saliency map weight, wherein the calculation formula is as follows:
• B=0.5*(0.5+I*(W1−W2)*0.5)+0.5*(0.5+V*(W2−W1)*0.5)
• wherein I and V represent the input infrared image and visible light image respectively, and W1 and W2 represent the salience weights obtained for the infrared image and visible light image respectively;
• 5) implementing the contrast-based pixel fusion strategy for the detail layer obtained after object difference, designing a sliding window to perform global sliding on the detail images of infrared and visible light respectively, comparing the pixel values of the corresponding detail images, and taking 1 if the values of eight neighborhoods of the current pixel of the infrared image are greater than those of eight neighborhoods of the corresponding pixel of the visible light;
• otherwise, taking 0;
• generating the corresponding binary weight map X according to the scanned sliding window;
• and then fusing the detail layer;
• D=D(I)*X+D(V)*(1−X)
• 6) linearly weighting the background layer and the detail layer to obtain:
• F=B+D
• wherein F represents the fusion result, and B and D represent the background layer fusion result and detail layer fusion result;
• 7) converting the color space:
• converting the fusion image back to the RGB image, and adding the hue and saturation previously retained;
• restoring to the RGB color space from the HSV color space by updating the V information saved into the fusion image in combination with the previously retained H and S;
• 8) enhancing the color:
• enhancing the color of the fusion image to generate a fusion image with higher resolution and contrast;
• and performing pixel-level image enhancement for the contrast of each pixel;
• performing color correction and enhancement on the restored image to generate a three-channel image that is consistent with observation and detection;
• and performing color enhancement on the R channel, G channel and B channel respectively to obtain the final fusion image.