US 11,388,336 B2Granted

Horizontal calibration method and system for panoramic image or video, and portable terminal

AssigneeArashi Vision Inc.

InventorsYuming Xie, Wenxue Peng

Life of this patent

Estimating the term…

Loading patent…

Abstract

The present invention provides a horizontal calibration method and system for a panoramic image or video, and a portable terminal. The method comprises: receiving images or video frames of two circles after being imaged by two cameras having opposite photographing directions; obtaining a horizontal calibration matrix during photographing; and pasting, according to the horizontal calibration matrix, the images or video frames of the two circles to a left hemispherical curved surface and a right hemispherical curved surface of a spherical panoramic mathematical model in a UV mapping manner, respectively so that the panoramic images or video frames pasted in the spherical panoramic mathematical model are horizontal. According to the present invention, when an image or a video frame photographed by a panoramic camera is not horizontal, the panoramic image or the video frame pasted in a spherical panoramic mathematical model is still horizontal.

Claims12

Dependent claims

A horizontal calibration method for a panoramic image or video, comprising steps of:

receiving images or video frames of two circles after being imaged by two cameras having opposite photographing directions;
acquiring a horizontal calibration matrix during photographing by means of a gyroscope;
and
mapping, according to the horizontal calibration matrix, the images or video frames of the two circles to a left hemispherical curved surface and a right hemispherical curved surface of a spherical panoramic mathematical model in a UV mapping manner, respectively, whereby the panoramic images or video frames mapped in the spherical panoramic mathematical model being horizontal;
wherein the step of acquiring the horizontal calibration matrix during photographing by means of a gyroscope, further comprises steps of:
during photographing, according to a gravity vector and a reference absolute horizontal gravity vector (0, 0, −1) of the gyroscope, acquiring a horizontal calibration quaternion, specifically:
calculating the horizontal calibration quaternion q from v1 rotation to v2, as:
v
⁢
⁢
1
=
[
x
⁢
⁢
1
y
⁢
⁢
1
z
⁢
⁢
1
]
v
⁢
⁢
2
=
[
x
⁢
⁢
2
y
⁢
⁢
2
z
⁢
⁢
2
]
v
⁢
⁢
3
=
[
x
⁢
⁢
3
y
⁢
⁢
3
z
⁢
⁢
3
]
dot
=
v
⁢
⁢
1
·
v
⁢
⁢
2
=
x
⁢
⁢
1
⁢
x
⁢
⁢
2
+
y
⁢
⁢
1
⁢
y
⁢
⁢
2
+
z
⁢
⁢
1
⁢
z
⁢
⁢
2
=

v
⁢
⁢
1

×

v
⁢
⁢
2

×
cos
⁡
(
angle
)
⁢
⁢
angle
=
a
⁢
⁢
cos
⁡
(
dot
)
*
180
/
π
⁢
⁢
v
⁢
⁢
3
=
v
⁢
⁢
1
×
v
⁢
⁢
2
=
[
x
⁢
⁢
1
y
⁢
⁢
1
z
⁢
⁢
1
]
×
[
x
⁢
⁢
2
y
⁢
⁢
2
z
⁢
⁢
2
]
=
[
y
⁢
⁢
1
⁢
z
⁢
⁢
2
-
z
⁢
⁢
1
⁢
y
⁢
⁢
2
z
⁢
⁢
1
⁢
x2
-
x
⁢
⁢
1
⁢
z
⁢
⁢
2
x
⁢
⁢
1
⁢
y2
-
y
⁢
⁢
1
⁢
x
⁢
⁢
2
]
⁢
⁢
q
=
[
w
x
y
z
]
=
[
cos
⁡
(
angle
/
180
*
π
/
2
)
sin
⁡
(
angle
/
180
*
π
/
2
)
*
x
⁢
⁢
3
sin
⁡
(
angle
/
180
*
π
/
2
)
*
y
⁢
⁢
3
sin
⁡
(
angle
/
180
*
π
/
2
)
*
z
⁢
⁢
3
]
where, v1 represents the gravity vector of the gyroscope, v2 represents the reference absolute horizontal gravity vector, angle represents a rotation angle about an axis, v3 represents a vector of multiplying matrix v1 by matrix v2;
and
converting the horizontal calibration quaternion to the horizontal calibration matrix, specifically, converting the horizontal calibration quaternion to the horizontal calibration matrix M0 of 4*4:
M
⁢
⁢
0
=
[
1.0
-
2.0
*
(
y
2
+
z
2
)
2.0
*
(
xy
+
wz
)
2.0
*
(
xz
-
wy
)
0
2.0
*
(
xy
-
wz
)
1.0
-
2.0
*
(
x
2
+
z
2
)
2.0
*
(
yz
+
wx
)
0
2.0
*
(
xz
+
wy
)
2.0
*
(
yz
-
wx
)
1.0
-
2.0
*
(
x
2
+
y
2
)
0
0
0
0
1
]
.

403 more elements

A horizontal calibration method for a panoramic image or video, comprising steps of:

receiving images or video frames of two circles after being imaged by two cameras having opposite photographing directions;
acquiring a horizontal calibration matrix during photographing by means of user manual calibration;
and
mapping, according to the horizontal calibration matrix, the images or video frames of the two circles to a left hemispherical curved surface and a right hemispherical curved surface of a spherical panoramic mathematical model in a UV mapping manner, respectively, whereby the panoramic images or video frames mapped in the spherical panoramic mathematical model being horizontal;
wherein the step of acquiring the horizontal calibration matrix during photographing by means of user manual calibration, further comprising steps of:
acquiring an Initial Quaternion q1 before user manual calibration,
q
⁢
⁢
1
=
[
w
x
y
z
]
;
receiving user's gesture operation instructions to a screen, and calculating a quaternion q3 after gesture operation based on a formula of a quaternion q2 constructed according to a rotation axis and a rotation angle about the axis;
where the formula of the quaternion q2 constructed according to a rotation axis and a rotation angle about the axis, is:
q
⁢
⁢
2
=
[
w
⁢
⁢
2
x
⁢
⁢
2
y
⁢
⁢
2
z
⁢
⁢
2
]
=
[
cos
⁡
(
angle
/
180
*
π
/
2
)
sin
⁡
(
angle
/
180
*
π
/
2
)
*
axis
·
x
sin
⁡
(
angle
/
180
*
π
/
2
)
*
axis
·
y
sin
⁡
(
angle
/
180
*
π
/
2
)
*
axis
·
z
]
,
where user's gesture operation instructions to a screen are about the world coordinate system x, y, and z axis to rotate with an angle:
−dy*180/320/8/180*π, −dx*180/320/8/180*π, deltaSlope, respectively;
here, dx represents a value of user horizontal sliding screen, dy represents a value of user longitudinal sliding screen, deltaSlope represents slope changing for user sliding operation with one finger, slope changing:
deltaSlope=a tan(currSlope)*180/π−a tan(preSlope)*180/π, a current Slope:
currSlope=(y2−y1)/(x2−x1), coordinates of a first point are p1(x1, y1) coordinates of a second point are p2(x2, y2), preslope represents slope changing for user previous sliding operation with one finger;
and
calculating the quaternion q3 after gesture operation based on the following formula:
q
⁢
⁢
3
=
q
⁢
⁢
1
×
q
⁢
⁢
2
=
[
w
*
w
⁢
⁢
2
-
x
*
x
⁢
⁢
2
-
y
*
y
⁢
⁢
2
-
z
*
z
⁢
⁢
2
w
*
x
⁢
⁢
2
+
x
*
w
⁢
⁢
2
+
y
*
z
⁢
⁢
2
-
z
*
y
⁢
⁢
2
w
*
y
⁢
⁢
2
+
y
*
w
⁢
⁢
2
+
z
*
x
⁢
⁢
2
-
x
*
z
⁢
⁢
2
w
*
z
⁢
⁢
2
+
z
*
w
⁢
⁢
2
+
x
*
y
⁢
⁢
2
-
y
*
x
⁢
⁢
2
]
;
and
converting the quaternion q3 after gesture operation to the horizontal calibration matrix, specifically, converting the horizontal calibration quaternion to the horizontal calibration matrix M0 of 4*4:
M
⁢
⁢
0
=
[
1.0
-
2.0
*
(
y
2
+
z
2
)
2.0
*
(
xy
+
wz
)
2.0
*
(
xz
-
wy
)
0
2.0
*
(
xy
-
wz
)
1.0
-
2.0
*
(
x
2
+
z
2
)
2.0
*
(
yz
+
wx
)
0
2.0
*
(
xz
+
wy
)
2.0
*
(
yz
-
wx
)
1.0
-
2.0
*
(
x
2
+
y
2
)
0
0
0
0
1
]
.

325 more elements

A horizontal calibration method for a panoramic image or video, comprising steps of:

receiving images or video frames of two circles after being imaged by two cameras having opposite photographing directions;
acquiring a horizontal calibration matrix during photographing by means of user manual calibration;
obtaining a spherical panoramic mathematical model after relative rotation via the horizontal calibration matrix being multiplied by the spherical panoramic mathematical model of longitude and latitude;
and
mapping the images or video frames of the two circles respectively to a left hemispherical curved surface and a right hemispherical curved surface of the spherical panoramic mathematical model after relative rotation in a UV mapping manner;
whereby the panoramic images or video frames mapped in the spherical panoramic mathematical model being horizontal;
wherein the step of obtaining a spherical panoramic mathematical model after relative rotation via the horizontal calibration matrix being multiplied by the spherical panoramic mathematical model of longitude and latitude, further comprises step of:
during generating data of UV mapping, according to calculation of the spherical panoramic mathematical model of longitude and latitude, obtaining each vertex of the spherical panoramic mathematical model based on the following formula:
V
=
[
x
y
z
1
]
=
[
sin
⁢
⁢
φcosθ
sin
⁢
⁢
φsinθ
cos
⁢
⁢
φ
1
]
,
where longitude φ has a degree range of 0-180 degrees, latitude θ has a degree range of 0 to 360 degrees;
multiplying the vertex by the horizontal calibration matrix according to the following formula:
V
=
[
x
y
z
1
]
×
M
⁢
⁢
0
=
[
x
′
y
′
z
′
1
]
;
and
converting the vertex multiplied by the horizontal calibration matrix back to longitude and latitude, and calculating the UV coordinates;
wherein if acquiring the horizontal calibration matrix during photographing by means of both a gyroscope and user manual calibration, then obtaining the vertex V multiplied by the horizontal calibration matrix via steps of:
acquiring an intermediate matrix by multiplying the vertex V by the horizontal calibration matrix obtained during photographing via user manual calibration, and then, multiplying the intermediate matrix by the horizontal calibration matrix obtained during photographing using a gyroscope, whereby acquiring the vertex V multiplied by the horizontal calibration matrix.

60 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/490,594Filed 2 Mar 2017
Published: US 2020/0007761 A12 Jan 2020
Granted: 12 Jul 2022
Term ends: Not estimatedThe source records don’t support an estimate.

Classification15 codes

G06T 7/70
Image analysis: determining position or orientation of objects or cameras
G02B 13/06
Optical objectives specially designed for the purposes specified below: panoramic objectives; So-called "sky lenses" including panoramic…
G06F 17/16
Complex mathematical operations: matrix or vector computation , e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
G06T 5/50
Image enhancement or restoration: using two or more images, e.g. averaging or subtraction