# US 2022/0044356 A1 · Large-field-angle image real-time stitching method based on calibration

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044356 A1
- Document ID: US20220044356A1
- Status: Published application
- Kind: A1
- Application number: 17/282,635
- Filed: 2020-03-05
- Published: 2022-02-10
- Earliest claimed: 2019-12-17
- Assignees: Dalian University of Technology
- Applicants: Dalian University of Technology
- Inventors: Wei ZHONG; Yuankai XIANG; Haojie LI; Zhihui WANG; Risheng LIU; Xin FAN; Zhongxuan LUO
- Web page: https://patentgrep.com/patent/US20220044356A1

## CPC

- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- G06T 5/40: using histogram techniques
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T 2207/10024: Color image
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20072: Graph-based image processing
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30208: Marker matrix
- Y02T 10/40: Engine management systems
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- Y02T: CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION

## Abstract

The present invention discloses a large-field-angle image real-time stitching method based on calibration, and belongs to the field of image processing and computer vision. First, a calibration algorithm is used to solve the positional relationship between cameras, and the prior information is used to solve a homography matrix between images. The system is easy to build, and the program is simple and easy to implement; an overlapping area ROI of images can be calculated by the homography matrix between images, and an energy model thereof can be built and solved with a graph cut algorithm; the graph cut algorithm has high time complexity and depends on the number of nodes in a graph; here, images are divided into layers, and solutions are obtained layer by layer and iterated, so as to find a local optimal solution that approximates a global optimal solution; and finally, a stitched image is further optimized by simple linear fusion of stitching seams and histogram equalization of the stitched image, which can make the vision thereof look more natural.

## Claims

### Claim 1

A large-field-angle image real-time stitching method based on calibration, comprising the following steps:
1) calibrating internal parameters and external parameters of each camera; the internal parameters include a focal length and an optical center; and the external parameters include rotation and translation;
2) calculating homography matrix: calculating a homography matrix between images according to the internal and external parameters of each camera, the placement angle between cameras, and a scene plane distance d;
3) building overlapping area model: using the homography matrix between images in step 2) to calculate an overlapping area ROI of images, and modeling the overlapping area;
4) dividing each image into blocks with a size of C1*C2, taking the divided blocks as nodes of a graph, performing graph cut to find a local optimal solution, then continuing to divide each node corresponding to an optimal stitching line corresponding to C1*C2 until a final block size is equal to a pixel value, and finally and approximately finding a global optimal solution by finding the local optimal solution each time;
5) conducting histogram equalization for exposure compensation; and
6) conducting histogram equalization to reduce image exposure difference.

### Claim 2 (depends on 1)

The large-field-angle image real-time stitching method based on calibration according to claim 1, wherein the calculation of the homography matrix in step 2) specifically comprises the following steps:
2-1) imaging a plane in a scene by two cameras, and assuming that the unit normal vector of the plane in the coordinate system of the first camera is N, the distance from the plane to the center i.e. coordinate origin, of the first camera is d, and an appropriate value is selected for d according to actual conditions, then the plane π can be expressed as:
NTX1=d
wherein the internal parameters K of each camera are known, X1 is the coordinate of a three-dimensional point PP in the coordinate system of the first camera, and the coordinate of the three-dimensional point PP in the coordinate system of the second camera is X2, then the relationship between the two is:
X
2
=
R
*
X
1
+
T
X
2
=
R
⁢
⁢
X
1
+
T
⁢
1
d
⁢
N
T
⁢
X
1
=
(
R
+
T
⁢
1
d
⁢
N
T
)
⁢
X
1
=
H
′
⁢
X
1
2-2) obtaining the homography matrix from the coordinate system of the first camera, and the homography matrix needs to be transformed into the coordinate system of the imaging plane:
x1KX1
x2=KX2
H=KH′K−1.

### Claim 3 (depends on 1)

The large-field-angle image real-time stitching method based on calibration according to claim 1, wherein the building of the mathematical model in step 3) specifically comprises the following steps:
3-1) for the pixels of two images in the overlapping area: calculating the second norm of the RGB pixels corresponding to the overlapping area of the two images, and building t-links; the calculation formulas of the second norm are as follows:
e(p,q)=∥∇p−∇p′∥+∥∇q−∇q′∥
∥∇p−∇p′∥=(Rp−Rp′)2+(Gp−Gp′)2+(Bp−Bp′)2
∥∇q−∇q′∥=(Rq−Rq′)2+(Gq−Gq′)2+(Bq−Bq′)2
wherein e(⋅) is a weight function, p is a source image, q is a target image, p is the pixel value of one point in the image p, p′ is the pixel value of a p adjacent point, q is the pixel value of one point in the target image, q′ is the pixel value of a q adjacent point, Rp is the value of R channel at p point, Rp′ is the value of R channel at p′ point, Gp is the value of G channel at p point, Gp′ is the value of G channel at p′ point, Bp is the value of B channel at p point, Bp′ is the value of B channel at p′ point, Rq is the value of R channel at q point, Rq′ is the value of R channel at q′ point, Gq is the value of G channel at q point, Gq′ is the value of G channel at q′ point, Bq is the value of B channel at q point, and Bq′ is the value of B channel at q′ point;
3-2) finding the optimal stitching line (i.e., stitching seam) for the built model: solving the stitching seam by graph cut;
giving a set of registered images represented by I0 and I1; letting P be the overlapping area, L={0,1} a represents a label set, wherein 0 is corresponding to I0, and 1 is corresponding to l1;
the stitching seam represents the pixels located in the overlapping area, which are respectively marked as 0 and 1 labels to minimize the energy;
an energy function is defined as:
E
⁡
(
f
)
=
∑
p
,
q
∈
N
⁢
⁢
S
p
,
q
⁡
(
l
p
,
l
q
)
+
∑
p
∈
P
⁢
D
P
⁡
(
l
p
)
wherein Sp,q is a smoothing term representing the cost of assigning a pair of pixels (p, q) in the overlapping area to (lp,lq), lp is a label assigned to the pixel p, lq is a label assigned to the pixel q, and Dp is a data term representing the cost of marking the pixel p in the overlapping area as lp;
3-3) conducting feathering fusion, wherein feathering is to process only the areas on both sides of the stitching line, and the formulas thereof are:
g
⁡
(
x
)
=
(
1
-
α
)
⁢
f
0
⁡
(
x
)
+
α
⁢
⁢
f
2
⁡
(
x
)
α
=
d
pro
-
d
dst
d
pro
wherein g(x) represents the pixel value after fusion, α is a weight coefficient, f0(x) is the pixel value of image 1, f1(x) is the pixel value of image 2, dpro represents the width of a processing area, and ddst represents the distance between the current pixel and the stitching seam in x direction.
