Multispectral camera dynamic stereo calibration algorithm based on saliency features

A multispectral camera dynamic stereo calibration algorithm based on saliency features, comprising the following steps:

• step 1:
• original image correction:
• conducting de-distortion and binocular correction on an original image according to internal parameters and original external parameters of the infrared camera and the visible light camera;
• step 2:
• detecting the saliency of the infrared image and the visible light image respectively based on a histogram contrast method;
• 2-1) calculating saliency of each color;
• and defining the saliency value of the pixel color as:
• S
• ⁡
• (
• I
• k
• )
• =
• S
• ⁡
• (
• c
• l
• )
• =
• ∑
• j
• =
• 1
• n
• ⁢
• f
• j
• ⁢
• D
• ⁡
• (
• c
• l
• ,
• c
• j
• )
• wherein n is the total number of the image colors;
• cj is the color value of the pixel point Ik;
• D(cl, cj) indicates a distance between two colors cl and cj in a color space;
• and fj is occurrence probability of color cj;
• in order to reduce the number of color types and reduce the calculation complexity, each color channel is quantized into several values;
• 2-2) smoothing the color saliency, wherein the formula of the smoothed color saliency is as follows:
• S
• ′
• ⁡
• (
• c
• )
• =
• 1
• (
• m
• -
• 1
• )
• ⁢
• T
• ⁢
• ∑
• i
• =
• 1
• m
• ⁢
• (
• T
• -
• D
• ⁡
• (
• c
• ,
• c
• i
• )
• )
• ⁢
• S
• ⁡
• (
• c
• i
• )
• wherein m=n/4;
• T=Σi=1m D(c, ci) is the sum of the distances between the color c and m nearest colors;
• 2-3) determining the saliency according to the colors on the image pixel points to obtain a saliency image;
• step 3:
• extracting feature points on the infrared image, the visible light image and the saliency images thereof;
• step 4:
• matching the feature points extracted in the previous step;
• 4-1) dividing the infrared image and the visible light image into m×n blocks respectively;
• and conducting steps 4-2) to 4-6) for each feature point pil of the infrared image;
• 4-2) finding a block bxi,yil corresponding to pil in the infrared image, wherein the block at the same position in the visible light image as block bxi,yil is bxi,yir a block set {Bxi,yir} has the same abscissa and ordinate as the block bxi,yir, and a feature point set is recorded as {Pir};
• assessing the degree of similarity between pixel points;
• if the degree of similarity is greater than a threshold t1, regarding the pixel points as rough matching points, and recording the set as {pir};
• otherwise, discarding the points and selecting a next feature point to repeat step 4-2);
• 4-3) if the maximum similarity sfirst and the second maximum similarity ssecond in pil and {pir} satisfy:
• F(sfirst,ssecond)≥t2
• reserving the matching, wherein a point pir having the maximum similarity in {pir} is taken as a matching point, t2 is a threshold and F(sfirst, ssecond) is used for describing a relationship between sfirst and ssecond;
• if the maximum similarity and the second maximum similarity do not satisfy the formula, discarding the points and selecting a next feature point to repeat step 4-2);
• after selection according to the rule, matching the corresponding feature point p′il of pir in the infrared image according to the steps 4-2) and 4-3);
• reserving the matching pil, pir if p′il=pil is satisfied;
• if p′il=pil is not satisfied, discarding the points and selecting a next feature point to repeat step 4-2);
• 4-4) based on the infrared image feature point pil=(xil,yil), conducting parabolic fitting to optimize an integer pixel feature point pir=(xir,yir) corresponding to the visible light image, to obtain a sub-pixel feature point p′ir=(xir+j*rx,yir+j*ry) corresponding to the visible light image, wherein j*rx is a sub-pixel offset in x direction and j*ry is a sub-pixel offset in y direction;
• 4-5) based on the integer pixel feature point pir=(xir, yir) corresponding to the visible light image, calculating the sub-pixel feature point p′il=(xil+j*lx,yir+j*ly) corresponding to the infrared image according to the method of 4-4), wherein j*lx is a sub-pixel offset in x direction and j*ly is a sub-pixel offset in y direction;
• 4-6) obtaining a final matching point pair of p′il, p′ir;
• and selecting a next infrared image feature point to repeat steps 4-2) to 4-6);
• step 5:
• judging a feature point coverage area:
• dividing the image into m*n grids;
• if the feature points cover all the grids, executing a next step;
• otherwise continuing to shoot the image and repeating step 1 to step 4;
• step 6:
• correcting the calibration result:
• using image coordinates of all the feature points to calculate the positional relationship between the two cameras after correction;
• and then superimposing with the original external parameters.