# US 2021/0342633 A1 · Method of merging images and data processing device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342633 A1
- Document ID: US20210342633A1
- Status: Published application
- Kind: A1
- Application number: 17/027,733
- Filed: 2020-09-22
- Published: 2021-11-04
- Earliest claimed: 2020-04-29
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: CHANGHUN SUNG; DEOKHWAN KIM; BYUNGDEOK KIM; TAEUI KIM
- Web page: https://patentgrep.com/patent/US20210342633A1

## CPC

- G06T 7/33: using feature-based methods
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 7/32: using correlation-based methods
- G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)}
- G06V 10/757: {Matching configurations of points or features}
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20221: Image fusion; Image merging
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Feature points included in input images are extracted and matching information indicating mapping relationships for feature points included in different input images is generated. A reference image is selected among the input images based on the matching information. Valid images are determined among the input images by excluding noise images from the input images based on the matching information. A two-dimensional bundle adjustment is performed to generate synchronized images by aligning the valid images to the reference image. A merged image is generated by merging the reference image and the synchronized images. Image merging performance is enhanced by selecting the reference image highly correlated with the other input images and estimating exact homography based on the reference image.

## Claims

### Claim 1

A method of merging images, comprising:
extracting feature points from input images and generating matching information indicating mapping relationships for feature points from different input images;
selecting a reference image from among the input images based on the matching information;
determining valid images from among the input images by excluding noise images based on the matching information;
performing a two-dimensional bundle adjustment to generate synchronized images by aligning the valid images with the reference image; and
generating a merged image by merging the reference image and the synchronized images.

### Claim 2 (depends on 1)

The method of claim 1, wherein the selecting of the reference image includes:
for each input image among the input images, determining a correlation value between feature points of each input image and feature points of other input images among the input images based on the matching information; and
selecting as the reference image an input image among the input images having a highest correlation value.

### Claim 3 (depends on 2)

The method of claim 2, wherein the correlation value is determined based on a Euclidian distance between two reprojected points obtained by aligning two corresponding feature points respectively included in two input images to a same two-dimensional plane.

### Claim 4 (depends on 3)

The method of claim 3, wherein the correlation value is determined based on a number of the corresponding points having a Euclidian distance less than a threshold distance.

### Claim 5 (depends on 1)

The method of claim 1, wherein selecting of the reference image includes:
determining a homography matrix indicating transformation between two input images based on the matching information of feature points included in the two input images;
determining a Euclidian distance between two reprojected points obtained by aligning two corresponding feature points respectively included in the two input images to a same two-dimensional plane using the homography matrix;
determining that the two corresponding feature points are a valid pair when a Euclidian distance between the two corresponding feature points is shorter than a threshold distance;
for each input image among the input images, determining a correlation value corresponding to a sum of numbers of valid pairs between feature points of the each input image and feature points of the other input images among the input images; and
selecting as the reference image and input image among the input images having a highest correlation value.

### Claim 6 (depends on 1)

The method of claim 1, wherein the determining of the valid images among the input images includes:
determining a reference correlation value between feature points of the reference image and feature points for each of the input images; and
determining whether each of the input images is the noise image or the valid image based on the reference correlation value.

### Claim 7 (depends on 6)

The method of claim 6, wherein the correlation value is determined based on a Euclidian distance between two reprojected points obtained by aligning two corresponding feature points respectively included in the reference image and the each one of the input images to a same two-dimensional plane.

### Claim 8 (depends on 7)

The method of claim 7, wherein the correlation value is determined based on a number of the corresponding feature points having a Euclidian distance less than a threshold distance.

### Claim 9 (depends on 1)

The method of claim 1, wherein the determining of the valid images from among the input images includes:
determining a homography matrix indicating transformation between the reference image and each one of the input images based on the matching information of feature points included in the reference image and feature points included in each one of the input images;
determining a Euclidian distance between two reprojected points obtained by aligning two corresponding feature points respectively included in the reference image and the each one of the input images to a same two-dimensional plane using the homography matrix;
determining whether feature points of each one of the input images fall within a valid range wherein a corresponding Euclidian distance is less than a threshold distance;
determining whether each one of the input images is a noise image when a number of feature points for each one of the input images falls within the valid range less than a threshold number; and
determining whether each one of the input images is a valid image when the number of feature points for each one of the input images falls within the valid range not less than the threshold number.

### Claim 10 (depends on 1)

The method of claim 1, wherein the performing of the two-dimensional bundle adjustment includes:
determining a cost function based on the matching information; and
aligning valid images to the reference image such that a value of the cost function is minimized.

### Claim 11 (depends on 10)

The method of claim 10, wherein the cost function is a reprojection error function corresponding to a sum of Euclidian distances between feature points of the reference image and reprojected points obtained by aligning feature points of valid images to a two-dimensional plane corresponding to the reference image.

### Claim 12 (depends on 10)

The method of claim 10, wherein determining the cost function includes:
setting, as independent parameters of the cost function, coordinates of feature points of the reference image and initial homography matrices indicating transformations between the reference image and the valid images.

### Claim 13 (depends on 12)

The method of claim 12, wherein aligning the valid images to the reference image includes:
determining bundle adjustment homography matrices corresponding to the initial homography matrices when the value of the cost function is minimized and bundle adjustment feature points corresponding to the feature points of the reference image when the value of the cost function is minimized.

### Claim 14 (depends on 13)

The method of claim 13, wherein the bundle adjustment homography matrices and the bundle adjustment feature points are determined by performing a partial differential operation with respect to the initial homography matrices and the feature points of the reference image.

### Claim 15 (depends on 13)

The method of claim 13, wherein the bundle adjustment homography matrices and the bundle adjustment feature points are determined using a Gauss-Newton algorithm or a gradient-descent algorithm.

### Claim 16 (depends on 13)

The method of claim 13, wherein performing the two-dimensional bundle adjustment further includes:
determining a correction matrix indicating transformation between the feature points of the reference image and the bundle adjustment feature points; and
generating the synchronized images by aligning the feature points of the valid images to a two-dimensional plane corresponding to the reference image using the correction matrix and the bundle adjustment homography matrices.

### Claim 17 (depends on 1)

The method of claim 1, wherein the input images are images respectively captured by a plurality of cameras.

### Claim 18 (depends on 1)

The method of claim 1, wherein the input images are images sequentially captured by a single camera.

### Claim 19

A method of merging images, comprising:
extracting feature points from input images and generating matching information indicating mapping relationships between feature points included in different input images;
for each input image among the input images, determining a correlation value between feature points of the each input image and feature points of other input images among the input images based on the matching information;
selecting, as a reference image, an input image among the input images having a highest correlation value;
determining valid images among the input images by excluding noise images from the input images based on the matching information;
determining a cost function based on the matching information such that coordinates of feature points of the reference image and homography matrices indicating transformations between the reference image and the valid images are set as independent parameters of the cost function;
determining optimal homography matrices and optimal feature points corresponding to a minimum value of the cost function by adjusting feature points of the reference image and the homography matrices;
determining a correction matrix indicating transformation between feature points of the reference image and the optimal feature points;
generating synchronized images by aligning feature points of the valid images to a two-dimensional plane corresponding to the reference image using the correction matrix and the optimal homography matrices; and
generating a merged image by merging the reference image and the synchronized images.

### Claim 20

A data processing device comprising:
a feature point generator configured to extract feature points from input images and generate matching information indicating mapping relationships between feature points included in different input images;
a reference image selector configured to select a reference image among the input images based on the matching information;
an image filter configured to determine valid images among the input images by excluding noise images from the input images based on the matching information;
a synchronizer configured to perform a two-dimensional bundle adjustment to generate synchronized images by aligning the valid images to the reference image; and
an image merger configured to generate a merged image by merging the reference image and the synchronized images.
