Bi-level optimization-based infrared and visible light fusion method

A bi-level optimization-based infrared and visible light fusion method, wherein the method comprises the following steps:

• 1) obtaining registered infrared and visible light images, and respectively calibrating each lens and jointly calibrating the respective systems of the visible light binocular camera and the infrared binocular camera;
• 1-1) respectively calibrating the infrared camera and the visible light camera by the Zhangzhengyou calibration method to obtain internal parameters including focal length and principal point position and external parameters including rotation and translation of each camera;
• 1-2) calculating the positional relationship of the same plane in the visible light image and the infrared image by using RT obtained by joint calibration and the detected checker corners, and registering the visible light image to the infrared image by using a homography matrix;
• 2) converting the color space of the visible light image from an RGB image to an HSV image, extracting the value information of the color image as the input of image fusion, and retaining the original hue and saturation;
• 3) carrying out bi-level paradigm-based mathematical modeling on the input infrared image and the visible light image with the color space converted;
• establishing two separate models, namely the upper-level sub-problem Leader and the lower-level sub-problem Follower to solve the same problem.
• Leader
• ⁢
• ⁢
• (
• l
• )
• :
• min
• F
• ⁢
• 
• F
• -
• I
• 
• 2
• 2
• +
• γ
• ⁢
• 
• F
• -
• V
• 
• 2
• 2
• Follower
• ⁢
• ⁢
• (
• f
• )
• :
• s
• .
• t
• .
• F
• ∈
• argmin
• F
• ⁢
• 
• ∇
• F
• -
• ∇
• I
• 
• 1
• +
• β
• ⁢
• 
• ∇
• F
• -
• ∇
• V
• 
• 1
• wherein F represents the fused image, the infrared image and the visible light image are respectively represented by I, V, ∇ represents the operator for obtaining the gradient, and γ, β represent parameters of Leader and Follower respectively;
• 4) solving the upper-level sub-problem to obtain the fusion result of the target in the image domain;
• the target result is obtained by solving the following formula:
• F
• 1
• k
• +
• 1
• =
• argmin
• F
• ⁢
• 
• F
• -
• I
• 
• 2
• 2
• +
• γ
• ⁢
• 
• F
• -
• V
• 
• 2
• 2
• wherein F1k+1 represents the result of the upper-level problem;
• and the goal has a simple closed-form solution and thus is directly obtained by the closed-form solution shown as the following formula:
• F
• 1
• k
• +
• 1
• =
• 1
• +
• γ
• ⁢
• V
• 1
• +
• α
• wherein F represents the fused image, I represents the infrared image, V represents the visible light image, γ represents the weight parameter of the visible light and infrared image, and α represents the weight variable parameter;
• 5) solving the lower-level sub-problem to obtain the fusion result of the target in the gradient domain;
• and using the alternating direction multiplier method to transform an unconstrained problem into a constrained problem by introducing auxiliary variables, and then solving the problem under the framework;
• 6) obtaining two estimates F1k+1, Ffk+1 of the fusion result under different characteristics by solving the upper-level and lower-lever sub-problems, and to fuse the two components into an image F, linearly combining the two components, which is expressed as the following form:
• Fk+1=αkFlk+1+(1−αk)Ffk+1
• wherein Fk+1 is the final result of each iteration and is manually selected based on experience;
• 7) converting the color space:
• converting the fusion image back to the RGB image, and adding the hue and saturation previously retained;
• restoring to the RGB color space from the HSV color space by updating the V information saved into the fusion image in combination with the previously retained H and S;
• 8) enhancing the color:
• enhancing the color of the fusion image to generate a fusion image with higher resolution and contrast;
• and performing pixel-level image enhancement for the contrast of each pixel;
• performing color correction and enhancement on the restored image to generate a three-channel image that is consistent with observation and detection;
• and performing color enhancement on the R channel, G channel and B channel respectively to obtain the final fusion image.