# US 2022/0044443 A1 · Fisheye camera calibration system, method and electronic device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044443 A1
- Document ID: US20220044443A1
- Status: Published application
- Kind: A1
- Application number: 17/505,658
- Filed: 2021-10-20
- Published: 2022-02-10
- Earliest claimed: 2019-05-22
- Assignees: Sichuan Visensing Technology Co., Ltd.
- Applicants: Sichuan Visensing Technology Co., Ltd.
- Inventors: Xianyu Su; Jia Ai
- Web page: https://patentgrep.com/patent/US20220044443A1

## CPC

- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- H04N 17/002: {for television cameras}
- G06T 3/047: Fisheye or wide-angle transformations
- G06T 5/80: Geometric correction
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- G06T 2207/30208: Marker matrix
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Provided are a fisheye camera calibration system, method and an electronic device. The system includes a polyhedral target, a fisheye camera and an electronic device. The polyhedral target includes an inner surface and multiple markers provided on the inner surface, and the inner surface is composed of multiple hexagonal and pentagonal planes. The fisheye camera is used for photographing the polyhedral target to collect a target image, where the polyhedral target and the multiple markers provided on the inner surface of the polyhedral target are captured in the target image. The electronic device is used for acquiring initial values of k1, k2, k3, k4, k5, u0, v0, mu, mv, Tj and Rj, and using a Levenberg-Marquardt algorithm to optimize the initial values of k1, k2, k3, k4, k5, u0, v0, mu, mv, Tj and Rj so as to determine imaging model parameters of the fisheye camera.

## Claims

### Claim 1

A fisheye camera calibration system, comprising:
a polyhedral target, comprising an inner surface and a plurality of markers disposed on the inner surface, the inner surface being composed of a plurality of hexagonal planes and a plurality of pentagonal planes;
a fisheye camera, configured to photograph the polyhedral target to capture a target image, wherein the polyhedral target and the plurality of markers disposed on the inner surface are captured in the target image; and
an electronic device, configured to:
fit, according to the target image, a selected radial distortion model with an equidistant projection model, to obtain initial values of distortion parameters k1, k2, k3, k4 and k5;
calculate a radius rmax according to the initial values of the distortion parameters k1, k2, k3, k4 and k5 and an equation rmax=k1θmax+k2θmax3+k3θmax5+k4θmax7+k5θmax9, where θmax represents a maximum field of view of the fisheye camera;
perform ellipse fitting on the polyhedral target in the target image, and obtain initial values of u0, v0, mu and mv according to the fitted ellipse and the value of rmax, where (u0, v0) represents a principal point, mu represents the number of pixels per unit distance in a horizontal direction of coordinates of the target image, and mv represents the number of pixels per unit distance in a vertical direction of coordinates of the target image;
acquire, according to the polyhedral target, a translation matrix Tj and a rotation matrix Rj of each of the pentagonal and hexagonal planes where the markers are located; and
optimize the initial values of k1, k2, k3, k4, k5, u0, v0, mu, mv, Tj and Rj by using a Levenberg-Marquardt algorithm, thereby determining imaging model parameters of the fisheye camera.

### Claim 2 (depends on 1)

The system as claimed in claim 1, wherein the polyhedral target comprises a housing, the inner surface is defined inside the housing, and the inner surface is a hemispherical-like inner surface formed by splicing the plurality of pentagonal planes and the plurality of hexagonal planes, vertexes formed through the splicing are located on a first virtual spherical surface, and a line connecting a center of sphere of the first virtual spherical surface and a center of each one of the hexagonal and pentagonal planes is perpendicular to the one of the hexagonal and pentagonal planes.

### Claim 3 (depends on 2)

The system as claimed in claim 2, wherein all the pentagonal planes of the inner surface are tangent to a second virtual spherical surface, all the hexagonal planes of the inner surface are tangent to a third virtual spherical surface, and a center of sphere of the second virtual spherical surface and a center of sphere of the third virtual spherical surface both coincide with the center of sphere of the first virtual spherical surface.

### Claim 4 (depends on 3)

The system as claimed in claim 3, wherein the plurality of hexagonal planes comprise 8 complete regular-hexagonal planes and 4 partial regular-hexagonal planes, the plurality of pentagonal planes comprise 4 complete regular-pentagonal planes and 4 partial regular-pentagonal planes, and the length of sides of the pentagonal planes is equal to the length of sides of the hexagonal planes; and
wherein each of the partial regular-pentagonal planes is a half regular-pentagonal plane, and each of the partial regular-hexagonal planes is a half regular-hexagonal plane.

### Claim 5 (depends on 2)

The system as claimed in claim 2, wherein the polyhedral target further comprises a marker layer containing the plurality of markers, the marker layer comprises a plurality of marker sub-layers, each of the marker sub-layers is provided with a plurality of markers of different sizes, and each of the marker sub-layers is attached onto one of the plurality of hexagonal and pentagonal planes; and
wherein a center of each of the marker sub-layers coincides with a center of the respective hexagonal or pentagonal plane, and one side of each of the marker sub-layers is parallel to one side of the respective hexagonal or pentagonal plane.

### Claim 6 (depends on 1)

The system as claimed in claim 1, wherein the plurality of markers are provided by a plurality of holes formed in the inner surface, the polyhedral target further comprises at least one light source disposed at an opposite side from the inner surface, and the plurality of markers are formed when light emitted from the at least one light source passes through the plurality of holes.

### Claim 7 (depends on 2)

The system as claimed in claim 2, wherein the plurality of pentagonal planes are a plurality of pentagonal metal sheets, and the plurality of hexagonal planes are a plurality of hexagonal metal sheets.

### Claim 8 (depends on 2)

The system as claimed in claim 2, wherein the fisheye camera is disposed at the center of sphere of the first virtual spherical surface.

### Claim 9 (depends on 2)

The system as claimed in claim 2, wherein the electronic device is further configured to:
select the radial distortion model r=k1θ+k2θ3+k3θ5+k4θ7+k5θ9 and the equidistant projection model r=fθ, where r represents a distance from a point in the target image to a distortion center, f represents a focal length of the fisheye camera, and θ represents an angle between an incident light and an optical axis of the fisheye camera;
fit the radial distortion model with the equidistant projection model, to obtain k1=f and k2=0, with the initial values of the distortion parameters k3, k4 and k5 set as 0; and
calculate the radius rmax as f*θmax, i.e., rmax=f*θmax, according to k1=f, k2=0, the initial values of k3, k4 and k5 that are set as 0, and the equation rmax=k1θmax+k2θmax3+k3θmax+k4θmax7+k5θmax9.

### Claim 10 (depends on 9)

The system as claimed in claim 9, wherein the electronic device is further configured to:
perform ellipse fitting on the polyhedral target in the target image, to obtain an ellipse fitting equation
(
u
-
u
0
a
)
2
+
(
v
-
ν
0
b
)
2
=
1
,
where a is a length of a major axis of an ellipse, and b is a length of a minor axis of the ellipse;
acquire boundary points of the polyhedral target in the target image, to obtain values of a and b;
obtain initial value of u0 and v0, according to the values of a and b and the ellipse fitting equation; and
obtain initial values of mu and mv, according to the values of a, b and rmax, and equations mu=a/rmax and mv=b/rmax.

### Claim 11 (depends on 5)

The system as claimed in claim 5, wherein the electronic device is further configured to:
obtain the translation matrix Tj and the rotation matrix Rj of each of the pentagonal and hexagonal planes relative to a world coordinate system, based on relationships that the vertexes of the spliced inner surface of the polyhedral target are located on the first virtual spherical surface, distances from the center of sphere of the first virtual spherical surface to the vertexes are equal, and the line connecting the center of sphere of the first virtual spherical surface to the center of each one of the pentagonal and hexagonal planes is perpendicular to the one of the pentagonal and hexagonal planes; and
obtain coordinates of each of the plurality of markers relative to the world coordinate system, according to the translation matrix Tj and the rotation matrix Rj of each of the pentagonal and hexagonal planes relative to the world coordinate system, and relationships that the center of each of the marker sub-layers coincides with the center of the respective pentagonal or hexagonal plane, and one side of each of the marker sub-layers is parallel to one side of the respective pentagonal or hexagonal plane,
wherein the world coordinate system is established according to the polyhedral target, and the origin of the world coordinate system is set at the center of sphere of the first virtual spherical surface.

### Claim 12 (depends on 2)

The system as claimed in claim 2, wherein the electronic device is further configured to:
minimize, by using the Levenberg-Marquardt algorithm, a sum of squares of differences between projection values and respective measurement values of the markers disposed on the polyhedral target, wherein the projection value of each marker refers to pixel coordinates corresponding to the marker that are calculated according to the projection model, and the measurement value of each marker refers to pixel coordinates corresponding to the marker in the target image under an image coordinate system.

### Claim 13

A fisheye camera calibration method, wherein the method is applied to a fisheye camera calibration system comprising a polyhedral target and a fisheye camera to be calibrated, the polyhedral target comprises an inner surface and a plurality of markers disposed on the inner surface, the inner surface is formed by splicing a plurality of hexagonal planes and a plurality of pentagonal planes, and the method comprises:
acquiring a target image captured by the fisheye camera, wherein the polyhedral target and the plurality of markers disposed on the inner surface are captured in the target image;
fitting, according to the target image, a selected radial distortion model with an equidistant projection model, to obtain initial values of distortion parameters k1, k2, k3, k4 and k5;
calculating a radius rmax according to the initial values of the distortion parameters k1, k2, k3, k4 and k5 and an equation rmax=k1θmax+k2θmax3+k3θmax5+k4θmax7+k5θmax9, where θmax represents a maximum field of view of the fisheye camera;
performing ellipse fitting on the polyhedral target in the target image, and obtaining initial values of u0, v0, mu and mv according to the fitted ellipse and the value of rmax, where (u0, v0) represents a principal point, mu represents the number of pixels per unit distance in a horizontal direction of coordinates of the target image, and mv represents the number of pixels per unit distance in a vertical direction of coordinates of the target image;
acquiring, according to the polyhedral target, a translation matrix Tj and a rotation matrix Rj of each of the pentagonal and hexagonal planes where the markers are located; and
optimizing the initial values of k1, k2, k3, k4, k5, u0, v0, mu, mv, Tj and Rj by using a Levenberg-Marquardt algorithm, thereby determining imaging model parameters of the fisheye camera.

### Claim 14 (depends on 13)

The method as claimed in claim 13, wherein the fitting, according to the target image, a selected radial distortion model with an equidistant projection model, to obtain initial values of distortion parameters k1, k2, k3, k4 and k5, comprises:
selecting a radial distortion model r=k1θ+k2θ3+k3θ5+k4θ7+k5θ9 and an equidistant projection model r=fθ, where r represents a distance from a point in the target image to a distortion center, f represents a focal length of the fisheye camera, and θ represents an angle between an incident light and an optical axis of the fisheye camera; and
fitting the radial distortion model with the equidistant projection model, to obtain k1=f and k2=0, with the initial values of the distortion parameters k3, k4 and k5 set as 0.

### Claim 15 (depends on 14)

The method as claimed in claim 14, wherein the calculating a radius rmax according to the initial values of the distortion parameters k1, k2, k3, k4 and k5 and an equation rmax=k1θmax+k2θmax3+k3θmax5+k4θmax7+k5θmax9, comprises:
calculating the radius rmax as f*θmax, i.e., rmax=f*θmax, according to k1=f, k2=0, the initial values of k3, k4 and k5 that are set as 0, and the equation rmax=k1θmax+k2θmax3+k3θmax5+k4θmax7+k5θmax9.

### Claim 16 (depends on 15)

The method as claimed in claim 15, wherein the performing ellipse fitting on the polyhedral target in the target image, and obtaining initial values of u0, v0, mu and mv according to the fitted ellipse and the value of rmax, comprises:
performing ellipse fitting on the polyhedral target in the target image, to obtain an ellipse fitting equation
(
u
-
u
0
a
)
2
+
(
v
-
v
0
b
)
2
=
1
,
where a major axis of an ellipse, and b is a length of a minor axis of the ellipse;
acquiring boundary points of the polyhedral target in the target image, to obtain values of a and b;
obtaining initial value of u0 and v0, according to the values of a and b and the ellipse fitting equation; and
obtaining initial values of mu and mv, according to the values of a, b and rmax, and equations mu=a/rmax and mv=b/rmax.

### Claim 17 (depends on 11)

The method as claimed in claim 11, wherein the method further comprises:
acquiring the initial values of mu and mv, according to parameters of a lens of the fisheye camera.

### Claim 18 (depends on 16)

The method as claimed in claim 16, wherein the polyhedral target further comprises a plurality of marker layers, each of the plurality of marker layers comprises a part of the plurality of markers, and each of the plurality of marker layers is attached onto one of the plurality of hexagonal and pentagonal planes;
the acquiring, according to the polyhedral target, a translation matrix Tj and a rotation matrix Rj of each of the pentagonal and hexagonal planes where the markers are located, comprises:
obtaining a translation matrix Tj and a rotation matrix Rj of each of the pentagonal and hexagonal planes relative to a world coordinate system, based on relationships that vertexes of the spliced inner surface of the polyhedral target are located on a first virtual spherical surface, distances from a center of sphere of the first virtual spherical surface to the vertexes are equal, and a line connecting the center of sphere of the first virtual spherical surface to a center of each one of the pentagonal and hexagonal planes is perpendicular to the one of the pentagonal and hexagonal planes; and
obtaining coordinates of each of the plurality of markers relative to the world coordinate system, according to the translation matrix Tj and the rotation matrix Rj of each of the pentagonal and hexagonal planes relative to the world coordinate system, and relationships that a center of each of the marker layers coincides with the center of the respective pentagonal or hexagonal plane, and one side of each of the marker layers is parallel to one side of the respective pentagonal or hexagonal plane,
wherein the world coordinate system is established according to the polyhedral target, and the origin of the world coordinate system is set at the center of sphere of the first virtual spherical surface.

### Claim 19 (depends on 18)

The method as claimed in claim 18, wherein the optimizing the initial values of k1, k2, k3, k4, k5, u0, v0, mu, mv, Tj and Rj by using a Levenberg-Marquardt algorithm, comprises:
minimizing, by using the Levenberg-Marquardt algorithm, a sum of squares of differences between projection values and respective measurement values of the markers disposed on the polyhedral target, wherein the projection value of each marker refers to pixel coordinates corresponding to the marker that are calculated according to the projection model, and the measurement value of each marker refers to pixel coordinates corresponding to the marker in the target image under an image coordinate system.

### Claim 20

An electronic device, comprising:
one or more processors;
a memory, electrically connected to the one or more processors; and
one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more application programs are configured to implement a method comprising:
acquiring a target image captured by a fisheye camera, wherein a polyhedral target and a plurality of markers disposed on an inner surface of the polyhedral target are captured in the target image, and the inner surface is formed by splicing a plurality of hexagonal planes and a plurality of pentagonal planes;
fitting, according to the target image, a selected radial distortion model with an equidistant projection model, to obtain initial values of distortion parameters k1, k2, k3, k4 and k5;
calculating a radius rmax according to the initial values of the distortion parameters k1, k2, k3, k4 and k5 and an equation rmax=k1θmax+k2θmax3+k3θmax5+k4θmax7+k5θmax9, where θmax represents a maximum field of view of the fisheye camera;
performing ellipse fitting on the polyhedral target in the target image, and obtaining initial values of u0, v0, mu and mv according to the fitted ellipse and the value of rmax, where (u0, v0) represents a principal point of the fisheye camera, mu represents the number of pixels per unit distance in a horizontal direction of coordinates of the target image, and mv represents the number of pixels per unit distance in a vertical direction of coordinates of the target image;
acquiring, according to the polyhedral target, a translation matrix Tj and a rotation matrix Rj of each of the pentagonal and hexagonal planes where the markers are located; and
optimizing the initial values of k1, k2, k3, k4, k5, u0, v0, mu, mv, Tj and Rj by using a Levenberg-Marquardt algorithm, thereby determining imaging model parameters of the fisheye camera.
