# US 12,112,490 B2 · Method and apparatus for stitching dual-camera images and electronic device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,490 B2
- Document ID: US12112490B2
- Status: Granted
- Kind: B2
- Application number: 17/277,657
- Filed: 2020-05-11
- Published: 2022-03-31
- Granted: 2024-10-08
- Earliest claimed: 2019-05-10
- Assignees: State Grid Zhejiang Electric Power Co., Ltd. Taizhou Power Supply Company; Taizhou Economic Development Zone Transportation Inspection Branch of Taizhou Hongda Electric Power Construction Co., Ltd.
- Applicants: State Grid Zhejiang Electric Power Co., Ltd. Taizhou Power Supply Company; Taizhou Economic Development Zone Transportation Inspection Branch of Taizhou Hongda Electric Power Construction Co., Ltd.
- Inventors: Jiandong Si; Jian Yang; Chen Zhou; Jianhua Lv; Guode Ying; Qian Yu; Qian Zhang; Huang Zhang; Huanghao Ying; Xiaofei Wang; Xi Yang; Weizhong Chen; Dongqi Lu; Ge Zhou; Jieli Bao; Huiyong Qiu
- Web page: https://patentgrep.com/patent/US12112490B2

## CPC

- G06T 7/33: using feature-based methods
- H04N 5/2628: {Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation}
- G06T 3/60: Rotation of whole images or parts thereof
- H04N 5/2624: {for obtaining an image which is composed of whole input images, e.g. splitscreen}
- H04N 23/45: for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
- G06T 2207/20221: Image fusion; Image merging
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- Y02T 10/40: Engine management systems
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- Y02T: CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION

## Abstract

Disclosed are apparatus, electronic device, and method for stitching dual-camera images. The method includes collecting two images having overlapping photographic content. The method includes extracting and matching feature points of the two images. The method includes roughly aligning the two images by using a homography matrix. The method includes accurately aligning the two images by using a least squares method to complete stitching the two images.

## Claims

### Claim 1

A method for stitching dual-camera images, comprising:
collecting two images having overlapping photographic content;
extracting and matching feature points of the two images;
roughly aligning the two images by using a homography matrix; and
accurately aligning the two images by using a least squares method to complete stitching the two images,
wherein accurately aligning the two images by using the least squares method comprises:
assuming that a feature point set of an image A of the two images is P, a feature point set of an image B of the two images is Q, N pairs of matching feature points exist between feature point set P and feature point set Q, and feature points pi(xi, yi) and qj(xj, yj) belong to feature point set P and feature point set Q respectively and match each other, then calculating an Euclidean distance between pi(xi, yi) and qj (xj, yj) by using the following formula:
distpq=∥pi−qj∥=√{square root over ((x1−xj)2+(yi−yj)2)} (10),
wherein i is an integer and satisfies 0<i≤N, and j is an integer and satisfies 0<j≤K, wherein K denotes a number of feature points in the feature point set Q;
calculating an error by using the following formula:
error=Σi=1N|(Rpi+T)−qj|2 (11),
wherein R denotes a rotation matrix and T denotes a translation matrix; and
finding an optimal solution to the formula (11) by using the least squares method,
wherein the optimal solution is the rotation matrix R and the translation matrix T with a smallest value of the error, and mapping the image A onto the image B through the rotation matrix R and the translation matrix T with the smallest value of the error.

### Claim 2 (depends on 1)

The method of claim 1, wherein the feature points are Scale Invariant Feature Transform (SIFT) feature points.

### Claim 3 (depends on 2)

The method of claim 2, wherein extracting and matching the feature points of the two images comprises:
finding a feature point in a scale space of each of the two images separately according to an image multi-scale space and a Gaussian convolution kernel;
determining a gradient direction and an amplitude of the feature point by using a gradient distribution characteristic of pixels in a neighborhood of the feature point;
performing statistics on gradient directions and amplitudes corresponding to the pixels in the neighborhood of the feature point by using a histogram; and
constructing a KD-Tree data structure for storing the feature points of the two images and matching the feature points of the two images.

### Claim 4 (depends on 1)

The method of claim 1, wherein roughly aligning the two images by using the homography matrix comprises:
with an image A of the two images as a reference image, randomly selecting M feature points ai from the image A, and finding M feature points bi in an image B of the two images that match the feature points ai in a one-to-one correspondence, wherein i is an integer and satisfies 0<i≤M;
calculating the homography matrix through the M feature points ai and the M feature points bi, projecting feature points in image A other than the feature points ai onto the image B through a space transformation of the homography matrix, and counting a number of accurate matching pairs between the feature points in the image A other than the feature points ai and feature points in the image B other than the feature points bi,
repeating the preceding steps multiple times to obtain a plurality of homography matrices, and selecting, from the plurality of homography matrices, one homography matrix having a largest number of accurate matching pairs as a target homography matrix; and
roughly aligning the two images by using the target homography matrix.

### Claim 5 (depends on 4)

The method of claim 4, wherein a number of times the homography matrix is repeatedly obtained is greater than or equal to 1000.

### Claim 6

An apparatus for stitching dual-camera images, comprising a processor and a memory, wherein the memory stores instructions executable by the processor, and the processor is configured to:
collect two images having overlapping photographic content;
extract and match feature points of the two images;
roughly align the two images by using a homography matrix; and
accurately align the two images by using a least squares method to complete stitching the two images,
wherein the processor is configured to:
assume that a feature point set of an image A of the two images is P, a feature point set of an image B of the two images is Q, N pairs of matching feature points exist between feature point set P and feature point set Q, and feature points pi(xi, yi) and qj (xj, yj) belong to feature point set P and feature point set Q respectively and match each other, then calculate an Euclidean distance between pi(xi, yi) and qj(xj, yj) by using the following formula:
distpq=∥pi−qj∥=√{square root over ((xi−xj)2+(yi−yj)2)} (10),
wherein i is an integer and satisfies 0<i≤N, and j is an integer and satisfies 0<j≤K, wherein K denotes a number of feature points in the feature point set Q;
calculating an error by using the following formula:
error=Σi=1N|(Rpi+T)−qj|2 (11)
wherein R denotes a rotation matrix and T denotes a translation matrix; and
finding an optimal solution to the formula (11) by using the least squares method,
wherein the optimal solution is the rotation matrix R and the translation matrix T with a smallest value of the error, and mapping the image A onto the image B through the rotation matrix R and the translation matrix T with the smallest value of the error.

### Claim 7 (depends on 6)

The apparatus of claim 6, wherein the feature points are Scale Invariant Feature Transform (SIFT) feature points.

### Claim 8 (depends on 7)

The apparatus of claim 7, wherein the processor is configured to:
find a feature point in a scale space of each of the two images separately according to an image multi-scale space and a Gaussian convolution kernel;
determine a gradient direction and an amplitude of the feature point by using a gradient distribution characteristic of pixels in a neighborhood of the feature point;
perform statistics on gradient directions and amplitudes corresponding to the pixels in the neighborhood of the feature point by using a histogram; and
construct a KD-Tree data structure for storing the feature points of the two images and match the feature points of the two images.

### Claim 9

A non-transitory computer-readable storage medium storing computer instructions for causing a processor to execute a method for stitching dual-camera images, comprising:
collecting two images having overlapping photographic content;
extracting and matching feature points of the two images;
roughly aligning the two images by using a homography matrix; and
accurately aligning the two images by using a least squares method to complete stitching the two images;
wherein accurately aligning the two images by using the least squares method comprises:
assuming that a feature point set of an image A of the two images is P, a feature point set of an image B of the two images is Q, N pairs of matching feature points exist between feature point set P and feature point set Q, and feature points pi(xi, yi) and qj(xj, yj) belong to feature point set P and feature point set Q respectively and match each other, then calculating an Euclidean distance between pi(xi, yi) and qj(xj, yj) by using the following formula:
distpq=∥pi−qj∥=√{square root over ((xi−xj)2+(yi−yj)2)} (10),
wherein i is an integer and satisfies 0<i≤N, and j is an integer and satisfies 0<j≤K, wherein K denotes a number of feature points in the feature point set Q;
calculating an error by using the following formula:
error=Σi=1N|(Rpi+T)−qj|2 (11),
wherein R denotes a rotation matrix and T denotes a translation matrix; and
finding an optimal solution to the formula (11) by using the least squares method, wherein the optimal solution is the rotation matrix R and the translation matrix T with a smallest value of the error, and mapping the image A onto the image B through the rotation matrix R and the translation matrix T with the smallest value of the error.

### Claim 10 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein the feature points are Scale Invariant Feature Transform (SIFT) feature points.

### Claim 11 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, wherein extracting and matching the feature points of the two images comprises:
finding a feature point in a scale space of each of the two images separately according to an image multi-scale space and a Gaussian convolution kernel;
determining a gradient direction and an amplitude of the feature point by using a gradient distribution characteristic of pixels in a neighborhood of the feature point;
performing statistics on gradient directions and amplitudes corresponding to the pixels in the neighborhood of the feature point by using a histogram; and
constructing a KD-Tree data structure for storing the feature points of the two images and matching the feature points of the two images.

### Claim 12 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein roughly aligning the two images by using the homography matrix comprises:
with an image A of the two images as a reference image, randomly selecting M feature points ai from the image A, and finding M feature points bi in an image B of the two images that match the feature points ai in a one-to-one correspondence, wherein i is an integer and satisfies 0<i≤M;
calculating the homography matrix through the M feature points ai and the M feature points bi, projecting feature points in image A other than the feature points ai onto the image B through a space transformation of the homography matrix, and counting a number of accurate matching pairs between the feature points in the image A other than the feature points ai and feature points in the image B other than the feature points bi,
repeating the preceding steps multiple times to obtain a plurality of homography matrices, and selecting, from the plurality of homography matrices, one homography matrix having a largest number of accurate matching pairs as a target homography matrix; and
roughly aligning the two images by using the target homography matrix.

### Claim 13 (depends on 12)

The non-transitory computer-readable storage medium of claim 12, wherein a number of times the homography matrix is repeatedly obtained is greater than or equal to 1000.