# US 12,002,248 B2 · Image splicing method, computer-readable storage medium, and computer device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,248 B2
- Document ID: US12002248B2
- Status: Granted
- Kind: B2
- Application number: 18/267,505
- Filed: 2021-12-16
- Published: 2024-02-08
- Granted: 2024-06-04
- Earliest claimed: 2020-12-16
- Assignees: Arashi Vision Inc.
- Applicants: Arashi Vision Inc.
- Inventors: Tan Su; Fei Gao
- Web page: https://patentgrep.com/patent/US12002248B2

## CPC

- G06V 10/16: {using multiple overlapping images; Image stitching}
- H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio
- G06T 3/08: Projecting images onto non-planar surfaces, e.g. geodetic screens
- G06T 3/12: Panospheric to cylindrical image transformations
- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- H04N 5/2628: {Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation}
- H04N 5/265: Mixing
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G06T 2207/20221: Image fusion; Image merging
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The present application is applicable to the field of image processing, and provides an image splicing method and apparatus, a computer-readable storage medium, a computer device, and cameras. The method comprises: calculating a first mapping table corresponding to a lens of each camera according to internal parameters of the lens of each camera, and according to the first mapping table, projecting images collected by the corresponding lens of each camera at the same time onto a unit sphere to obtain multiple first spherical images corresponding to the lens of each camera; according to external parameters of the lens of each camera and the distance from the centers of circles formed by optical centers of the lens of multiple cameras and a preset plane above or below the circles, calculating a second mapping table corresponding to the lens of each camera, mapping the multiple first spherical images to a unified unit sphere according to the second mapping table, and splicing the first spherical images to obtain a panoramic spherical image. The present application can obtain more natural and realistic picture spliced at the top and bottom.

## Claims

### Claim 1

An image splicing method for panoramic spherical image, wherein the method comprises:
acquiring internal parameters and external parameters of lens of multiple cameras, a distance from a center of circle formed by an optical center of lens of the multiple cameras to a preset plane above or below the circle, and multiple images collected by the multiple cameras respectively, wherein the multiple images are images collected by the lens of the multiple cameras at the same time respectively;
according to the internal parameters of the lens of each camera, calculating a first mapping table corresponding to the lens of each camera; and according to the first mapping table corresponding to the lens of each camera, projecting images collected by the corresponding lens of each camera at the same time onto a unit sphere to obtain multiple first spherical images respectively corresponding to the lens of each camera; and
according to the external parameters of the lens of each camera and the distance from the center of circle formed by the optical center of the lens of the multiple cameras to a preset plane above or below the circle, calculating a second mapping table corresponding to the lens of each camera, and mapping the multiple first spherical images respectively corresponding to the lens of each camera to a unified unit sphere to obtain a panoramic spherical image by splicing, according to the second mapping table corresponding to the lens of each camera.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the unit sphere is specifically a unit sphere with the optical center of the lens serving as the center of sphere, and the unified unit sphere is specifically a unit sphere with the centers of circles formed by the optical centers of the lens of the multiple cameras serving as the center of sphere.

### Claim 3 (depends on 1)

The method according to claim 1, wherein after the step of obtaining a panoramic spherical image by splicing, the method further comprises:
projecting the panoramic spherical image to a plane according to a plane projection mode to obtain a planar image.

### Claim 4 (depends on 3)

The method according to claim 3, wherein after the step of projecting the panoramic spherical image to a plane according to a plane projection mode to obtain a planar image, the method further comprises: synthesizing the planar image into a video.

### Claim 5 (depends on 3)

The method according to claim 3, wherein the plane projection mode comprises an equidistant cylindrical projection method and a cubic projection method.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the step of calculating a second mapping table corresponding to the lens of each camera according to the external parameters of the lens of each camera and the distance from the center of circle formed by the optical center of the lens of the multiple cameras to a preset plane above or below the circle is specifically implemented according to the following equations:
p
i
⁢
0
s
=
R
i
·
p
i
s
^
=
[
p
i
⁢
0
⁢
x
s
,
p
i
⁢
0
⁢
y
s
,
p
i
⁢
0
⁢
z
s
]
,
p
i
⁢
1
s
=
{
d
1
/
p
i
⁢
0
⁢
z
s
·
p
i
⁢
0
s
-
c
i
,
p
i
⁢
0
⁢
z
s
≥
0
d
2
/-
p
i
⁢
0
⁢
z
s
·
p
i
⁢
0
s
-
c
i
,
p
i
⁢
0
⁢
z
s
<
0
,
p
i
s
=
f
2
⁢
p
i
s
ˆ
)
=
p
i
⁢
1
s
/
❘
"\[LeftBracketingBar]"
p
i
⁢
1
s
❘
"\[RightBracketingBar]"
,
wherein f2(piŝ) is the second mapping table which represents the mapping of three-dimensional coordinates piŝ of of a pixel point on a first spherical image of a lens of the ith camera to three-dimensional coordinates pis of the pixel point on the unified unit sphere, 1≤i≤n, n is the number of lens of the camera, Ri is a rotation matrix of the lens of the ith camera relative to a reference lens, and ci is the position of the lens of the ith camera relative to the centers of circles formed by the optical centers of the lens of the multiple cameras, which is expressed in three-dimensional coordinates; d1 is the distance from the centers of circles formed by the optical centers of the lens of the multiple cameras to a preset plane above the circles, and d2 is the distance from the centers of circles formed by the optical centers of the lens of the multiple cameras to a preset plane below the circles, pi0s is a obtained point after rotating and transforming the three-dimensional coordinates piŝ of a pixel point on a first spherical image using Ri, and the three-dimensional coordinates of pi0s are represented by [pi0xs, pi0ys, pi0zs].

### Claim 7 (depends on 1)

The method according to claim 1, wherein if the multiple cameras are placed in a room, then the distance from the centers of circles formed by the optical centers of the lens of the multiple cameras to a preset plane above the circles is the distance from the centers of circles to a top of the room, while the distance from the centers of circles formed by the optical centers of the lens of the multiple cameras to a preset plane below the circles is the distance from the centers of circles to a bottom of the room.

### Claim 8 (depends on 1)

The method according to claim 1, wherein the distance from the centers of circles formed by the optical centers of the lens of the multiple cameras to a preset plane above or below the circles is specifically acquired by the following steps:
acquiring the distance from the centers of circles formed by the optical centers of the lens of the multiple cameras to a preset plane above or below the circles by using a distance sensor; or automatically estimating the distance from the centers of circles formed by the optical centers of the lens of the multiple cameras to a preset plane above or below the circles.

### Claim 9 (depends on 8)

The method according to claim 8, wherein the step of automatically estimating the distance from the centers of circles formed by the optical centers of the lens of the multiple cameras to a preset plane above or below the circles specifically comprises the following operations:
selecting multiple distances within a preset distance range, wherein the multiple distances are distances from the centers of circles formed by the optical centers of the lens of the multiple cameras to multiple preset planes above or below the circles, and for each of the distances selected, respectively calculating a second mapping table corresponding to the lens of each camera according to the external parameters of the lens of each camera and each of the distances; mapping multiple first spherical images respectively corresponding to the lens of each camera to the unified unit sphere according to the second mapping table corresponding to the lens of each camera to obtain multiple unified unit spherical images respectively corresponding to the lens of each camera;
for each of the unified unit spherical images corresponding to the lens of each camera, projecting a part of the upper hemisphere of the unified unit spherical image to a plane which is at the selected distance from the centers of circles formed by the optical centers of the lens of the multiple cameras and above the circles according to the plane projection mode so as to obtain top projection pictures, and projecting a part of the lower hemisphere of the unified unit spherical image to a plane which is at the selected distance from the centers of circles formed by the optical centers of the lens of the multiple cameras and below the circles according to the plane projection mode so as to obtain bottom projection pictures; and
calculating an overall alignment error of the top projection pictures or the bottom projection pictures; and taking the distance corresponding to the minimum overall alignment error of the top projection pictures or the bottom projection pictures as the distance from the centers of circles formed by the optical centers of the lens of the multiple cameras to a preset plane above or below the circles.

### Claim 10 (depends on 9)

The method according to claim 9, wherein the step of selecting multiple distances within a preset distance range specifically comprises:
letting the vertical viewing angle of the lens be a, and letting the average distance from the lens of each camera to the centers of circles formed by the optical centers of the lens of the multiple cameras be r, and the number of different distances which are trialed in order of size is m, wherein m is an integer greater than or equal to 2, and the calculation method for the jth distance dj trialed is as follows:
dj=r·arc tan(2/(α−π)·m/j),j∈[1,m].

### Claim 11 (depends on 9)

The method according to claim 9, wherein the step of calculating an overall alignment error of the top projection pictures or the bottom projection pictures specifically comprises:
for the lens of each camera, respectively calculating an average pixel error within the overlapping area between the top projection pictures or the bottom projection pictures corresponding to the lens of two cameras adjacent to the lens of each camera;
calculating the sum of the average pixel errors of the lens of all the cameras as the overall alignment error of the top projection pictures or the bottom projection pictures.

### Claim 12 (depends on 11)

The method according to claim 11, wherein the step of calculating the overall alignment error of the top projection pictures or the bottom projection pictures is specifically implemented by the following equation:
E(dj)=ΣinD(Pi,j, Pi+1,j), wherein 1≤i≤n, n represents the number of the lens of the cameras, E (dj) represents the overall alignment error of the top projection pictures or the bottom projection pictures at the jth distance di trialed, D(Pi,j, Pi+1,j) represents the average pixel error at the jth distance dj trialed within the overlapping area between the top projection pictures or the bottom projection pictures corresponding to the lens of the ith camera and the lens of the (i+1)th camera which are adjacent to each other.

### Claim 13 (depends on 9)

The method according to claim 9, wherein after the step of taking the distance corresponding to the minimum overall alignment error of the top projection pictures or the bottom projection pictures as the distance from the centers of circles formed by the optical centers of the lens of the multiple cameras to a preset plane above or below the circles, the method further comprises the following operations:
according to the estimated distance from the centers of circles formed by the optical centers of the lens of the multiple cameras to a preset plane above or below the circles, controlling to adjust the distance from the lens of each camera to the preset plane so that the alignment error between the lens of each camera and the lens of two cameras adjacent thereto is minimum, and taking the distance from the lens of the camera to the preset plane when the alignment error between the lens of each camera and the lens of two cameras adjacent thereto is minimum as the final distance from the lens of the camera to a preset plane above or below the circle;
calculating a third mapping table corresponding to the lens of each camera respectively according to the external parameters of the lens of each camera and the final distance from the lens of the camera to a preset plane above or below the circle;
the operation of splicing the first spherical images to obtain a panoramic spherical image specifically comprises: mapping multiple first spherical images respectively corresponding to the lens of each camera to a unified unit sphere according to the third mapping table corresponding to the lens of each camera, and splicing the first spherical images to obtain a panoramic spherical image.

### Claim 14 (depends on 13)

The method according to claim 13, wherein the step of controlling to adjust the distance from the lens of each camera to the preset plane according to the estimated distance from the centers of circles formed by the optical centers of the lens of the multiple cameras to a preset plane above or below the circles so that the alignment error between the lens of each camera and the lens of two cameras adjacent thereto is minimum specifically comprises:
respectively calculating a second mapping table corresponding to the lens of each camera according to the external parameters of the lens of each camera and the estimated distance from the centers of circles formed by the optical centers of the lens of the multiple cameras to a preset plane above or below the circles;
mapping multiple first spherical images respectively corresponding to the lens of each camera to an unified unit sphere according to the second mapping table corresponding to the lens of each camera to obtain multiple unified unit spherical images respectively corresponding to the lens of each camera;
for each of the unified unit spherical images corresponding to the lens of each camera, projecting a part of the upper hemisphere of the unified unit spherical image to a plane which is at the selected distance from the centers of circles formed by the optical centers of the lens of the multiple cameras and above the circles according to the plane projection mode so as to obtain top projection pictures, and projecting a part of the lower hemisphere of the unified unit spherical image to a plane which is at the selected distance from the centers of circles formed by the optical centers of the lens of the multiple cameras and below the circles according to the plane projection mode so as to obtain bottom projection pictures;
according to the top projection pictures or the bottom projection pictures corresponding to the lens of each camera, calculating an alignment error between the lens of each camera and the lens of two cameras adjacent thereto, and sorting the lens of all the cameras in a descending order according to the alignment error;
starting from the lens of the camera with the maximum alignment error, adjusting the height of the lens in the vicinity of the estimated distance from the centers of circles formed by the optical centers of the lens of the multiple cameras to a preset plane above or below the circles until the alignment error is minimum;
processing the lens of each camera sequentially until the height adjustment of the lens of all the cameras is completed.

### Claim 15 (depends on 14)

The method according to claim 14, wherein the step of calculating an alignment error between the lens of each camera and the lens of two cameras adjacent thereto according to the top projection pictures or the bottom projection pictures corresponding to the lens of each camera specifically comprises:
calculating an average pixel error within the overlapping area between the top projection pictures or the bottom projection pictures corresponding to the lens of each camera and the lens of two cameras adjacent thereto.

### Claim 16 (depends on 14)

The method according to claim 14, wherein the step of adjusting the height of the lens in the vicinity of the estimated distance from the centers of circles formed by the optical centers of the lens of the multiple cameras to a preset plane above or below the circles specifically comprises:
recording the estimated distance from the centers of circles formed by the optical centers of the lens of the multiple cameras to a preset plane above or below the circles as d, adjusting the distance di from each of the lens of each camera to a preset plane above or below the circle, letting a point located right above or right below the circle on the preset plane be q, letting the coordinate of the lens of the ith camera be ci, letting the centers of circles formed by the optical centers of the lens of the multiple cameras be an origin O, and letting ∠qciO be a top angle β, β=arctan (d/ri), wherein ri is the distance from the lens of the ith camera to the centers of circles, that is, ri=|ci|, and a new top angle βj is trialed within the range of 0.9β to 1.1β to obtain a new distance dij=tan (βj)·ri from each of the lens of each camera to a preset plane above or below the circles.

### Claim 17 (depends on 13)

The method according to claim 13, wherein the step of calculating a third mapping table corresponding to the lens of each camera according to the external parameters of the lens of each camera and the final distance from the lens of the camera to a preset plane above or below the circle is specifically implemented according to the following equations:
p
i
⁢
0
s
=
R
i
·
p
i
s
^
=
[
p
i
⁢
0
⁢
x
s
,
p
i
⁢
0
⁢
y
s
,
p
i
⁢
0
⁢
z
s
]
,
p
i
⁢
1
s
=
{
d
i
⁢
1
/
p
i
⁢
0
⁢
z
s
·
p
i
⁢
0
s
⁢
c
i
,
p
i
⁢
0
⁢
z
s
≥
0
d
i
⁢
2
/-
p
i
⁢
0
⁢
z
s
·
p
i
⁢
0
s
-
c
i
,
p
i
⁢
0
⁢
z
s
<
0
,
p
i
s
=
f
3
(
p
i
s
ˆ
)
=
p
i
⁢
1
s
/
❘
"\[LeftBracketingBar]"
p
i
⁢
1
s
❘
"\[RightBracketingBar]"
wherein f3(piŝ) is the third mapping table which represents the mapping of three-dimensional coordinates piŝ of a pixel point on a first spherical image of a lens of the ith camera to three-dimensional coordinates pis of the pixel point on the unified unit sphere, 1≤i≤n, n is the number of lens of the camera, Ri is a rotation matrix of the lens of the ith camera relative to a reference lens, and ci is the position of the lens of the ith camera relative to the centers of circles formed by the optical centers of the lens of the multiple cameras, which is expressed in three-dimensional coordinates; di1 is the final distance from the lens of the ith camera to a preset plane above the circles, and di2 is the final distance from the lens of the ith camera to a preset plane below the circles, pi0s is a obtained point after rotating and transforming the three-dimensional coordinates piŝ of a pixel point on a first spherical image using Ri, and the three-dimensional coordinates of pi0s are represented by [pi0xs, pi0ys, pi0zs].

### Claim 18 (depends on 1)

A non-volatile computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the image splicing method for panoramic spherical image according to claim 1.

### Claim 19 (depends on 1)

A computer device, comprising:
one or more processors;
a memory; and
one or more computer programs, the processors and the memory being connected through a bus, wherein the one or more computer programs are stored in the memory and are configured to be executed by the one or more processors, wherein the processor, when executing the computer programs, implements the steps of the image splicing method for panoramic spherical image according to claims 1.