US 2020/0027243 A1 · APPARATUS FOR PROVIDING CALIBRATION DATA, CAMERA SYSTEM AND METHOD FOR OBTAINING CALIBRATION DATA

Apparatus comprising:

a first interface for receiving a plurality of partially overlapping images of an object from a corresponding plurality of camera positions being arranged along a first and a second direction according to a two-dimensional or three-dimensional camera pattern, wherein patterns comprising camera positions that differ in two directions are two-dimensional patterns and patterns comprising camera positions that differ in three directions are three-dimensional patterns;
an analyzing unit configured for selecting at least one corresponding reference point in an overlap area of a set of overlapping images, and for determining a displacement information along the first and the second direction of the reference point in each of the other images of the set of images, wherein a misalignment of the plurality of images along the first and the second direction is compensated by the displacement information so as to acquire aligned images;
a determining unit configured for determining an offset information between principal points at the plurality of camera positions using at least three aligned images;
and
a second interface for providing calibration data based on the displacement information and based on the offset information, the calibration data allowing for calibrating the plurality of images so as to comply to the camera pattern;
wherein the analyzing unit is configured to determine the displacement information minimizing an error of a first minimization criteria, wherein for a 2D camera pattern the first minimization criteria is based on the determination rule;
arg


min
x
1
,
x
2
,
x
3
,
…
,
x
N

∑
(
p
,
q
)
∈
Ω


v

(
H

(
x
p
,
C
p
,
C
q
)
·
m
p
,
q
(
p
)
)
-
v

(
H

(
x
q
,
C
p
,
C
q
)
·
m
p
,
q
(
q
)
)


with



x

=
∑
l

x
l
2

wherein the determining unit is configured to determine the offset information minimizing an error of a second minimization criteria, wherein for the 2D camera pattern the second minimization criteria is based on the determination rule
arg


min
x
1
,
x
2
,
x
3
,
…
,
x
N

∑
(
p
,
q
)
∈
Ψ



(
u

(
H

(
x
p
,
C
p
,
C
q
)
·
m
p
,
q
,
r
(
p
)
)
-
u

(
H

(
x
q
,
C
p
,
C
q
)
·
m
p
,
q
,
r
(
q
)
)
)

1
d

(
C
p
-
C
q
)
-
(
u

(
H

(
x
p
,
C
p
,
C
r
)
·
m
p
,
q
,
r
(
p
)
)
-
u

(
H

(
x
r
,
C
p
,
C
r
)
·
m
p
,
q
,
r
(
q
)
)
)

1
d

(
C
p
-
C
r
)


with



x

=
∑
l

x
l
2
wherein argmin denotes the minimization criteria and p, q and r denote indices of individual cameras. Cp, Cq, Cr denote camera positions of cameras p, q and r, d(Cp−Cq) is a distance between the camera positions, xp, xq and xr are vectors with parameters to be determined so as to minimizing the error for camera p, q and r, H(⋅) is a function providing a homography matrix undistorting points according to a parameter vector xp and camera positions Cp and Cq, mp,q=[m(p)p,q m(q)p,q] denotes a set of reference points wherein m(p)p,q corresponds to the respective reference points in camera p, mp,q,r=[m(p)p,q,r m(q)p,q,r m(r)p,q,r] denotes a set of reference points wherein m(p)p,q,r corresponds to the respective reference points in camera p, Functions u(⋅) and v(⋅) extract the horizontal/vertical component of an image point respectively, the set Ω comprises all pairs of partially overlapping images;
the set ψ comprises triplets of partially overlapping images;
such that each of the N cameras considered in the minimization problem is comprised in at least one pair of cameras comprised in Ω and at least in one triplet of cameras comprised in ψ;
and/or
wherein the analyzing unit is configured to determine the displacement information minimizing an error of a linearized first minimization criteria, wherein for a 2D camera pattern the linearized first minimization criteria is based on the determination rule;

arg


min
x
1
,
x
2
,
x
3
,
…
,
x
N

∑
(
p
,
q
)
∈
Ω

[
J
p
,
q
(
p
)

|

J
p
,
q
(
q
)
]
·
[
x
p
T
x
q
T
]
+
b
p
,
q



wherein

(
v

(
H

(
x
p
,
C
p
,
C
q
)
·
m
p
,
q
(
p
)
)
-
v

(
H

(
x
q
,
C
p
,
C
q
)
·
m
p
,
q
(
q
)
)
)
2
≈

[

J
p
,
q
(
p
)

|

J
p
,
q
(
q
)
]
·
[
x
p
T
x
q
T
]
+
b
p
,
q
wherein the determining unit is configured to determine the offset information minimizing an error of a linearized second minimization criteria, wherein for the 2D camera pattern the linearized second minimization criteria is based on the determination rule

arg


min
x
1
,
x
2
,
x
3
,
…
,
x
N

∑
(
p
,
q
,
r
)
∈
Ψ

[
J
p
,
q
,
r
(
p
)
J
p
,
q
,
r
(
q
)
J
p
,
q
,
r
(
r
)
]
·
[
x
p
T
x
q
T
x
r
T
]
+
b
p
,
q
,
r

wherein

(
(
u

(
H

(
x
p
,
C
p
,
C
q
)
·
m
p
,
q
,
r
(
p
)
)
-
u

(
H

(
x
q
,
C
p
,
C
q
)
·
m
p
,
q
,
r
(
q
)
)
)

1
d

(
C
p
-
C
q
)
-
(
u

(
H

(
x
p
,
C
p
,
C
r
)
·
m
p
,
q
,
r
(
p
)
)
-
u

(
H

(
x
r
,
C
p
,
C
r
)
·
m
p
,
q
,
r
(
q
)
)
)

1
d

(
C
p
-
C
r
)
)
2
≈
[

J
p
,
q
,
r
(
p
)
J
p
,
q
,
r
(
q
)
J
p
,
q
,
r
(
r
)
]
·
[
x
p
T
x
q
T
x
r
T
]
+
b
p
,
q
,
r
wherein argmin denotes the minimization criteria p, q and r denote indices of individual cameras, Cp, Cq, Cr denote camera positions of cameras p, q and r, d(Cp−Cq) is a distance between the camera positions, H(⋅) is a function providing a homography matrix undistorting points according to a parameter vector xp and camera positions Cp and Cq, corresponding to camera p with parameter vector xp, xq and xr, are vectors with parameters to be determined so as to minimizing the error for camera p, q and r, mp,q=[m(p)p,q m(q)p,q] denotes a set of reference points wherein m(p)p,q corresponds to the respective reference points in camera p, mp,q,r=[m(p)p,q,r m(q)p,q,r m(r)p,q,r] denotes a set of reference points wherein m(p)p,q,r corresponds to the respective reference points in camera p, the set Ω comprises all pairs of partially overlapping images;
the set ψ comprises triplets of partially overlapping images;
such that each of the N cameras considered in the minimization problem is comprised in at least one pair of cameras comprised in Ω and in at least one triplet of cameras comprised in ψ, functions u(⋅) and v(⋅) extract the horizontal/vertical component of an image point respectively, Jp,q=[J(p)p,q J(q)p,q] and Jp,q,r=[J(p)p,q,r J(q)p,q,r J(r)p,q,r] denote the Jacobi matrices for camera pairs and camera triplets, respectively, J(p)p,q and J(p)p,q,r denote sub-matrices corresponding to a camera p, bp,q and bp,q,r denote residual elements.

704 more elements

Method for acquiring calibration data, the method comprising:

receiving a plurality of partially overlapping images of an object from a corresponding plurality of camera positions being arranged along a first and a second direction according to a two-dimensional or three-dimensional camera pattern, wherein patterns comprising camera positions that differ in two directions are two-dimensional patterns and patterns comprising camera positions that differ in three directions are three-dimensional patterns;
selecting at least one corresponding reference point in an overlap area of a set of overlapping images, and determining a displacement information along the first and the second direction of the reference point in each of the other images of the set of images, such that a misalignment of the plurality of images along the first and the second direction is compensated by the displacement information so as to acquire aligned images;
determining an offset information between principal points at the plurality of camera positions using at least three aligned images;
and
providing calibration data based on the displacement information and based on the offset information, the calibration data allowing for calibrating the plurality of images so as to comply to the camera pattern;
and
determining the displacement information minimizing an error of a first minimization criteria, wherein for a 2D camera pattern the first minimization criteria is based on the determination rule;
argmin
x
1
,
x
2
,
x
3
,
…
,
x
N

∑
(
p
,
q
)
∈
Ω


v

(
H

(
x
p
,
C
p
,
C
q
)
·
m
p
,
q
(
p
)
)
-
v

(
H

(
x
q
,
C
p
,
C
q
)
·
m
p
,
q
(
q
)
)



with




x


=
∑
l

x
l
2
determining the offset information minimizing an error of a second minimization criteria, wherein for the 2D camera pattern the second minimization criteria is based on the determination rule
argmin
x
1
,
x
2
,
x
3
,
…
,
x
N

∑
(
p
,
q
,
r
)
∈
Ψ


(
u

(
H

(
x
p
,
C
p
,
C
q
)
·
m
p
,
q
,
r
(
p
)
)
-
u

(
H

(
x
q
,
C
p
,
C
q
)
·
m
p
,
q
,
r
(
q
)
)
)

1
d

(
C
p
-
C
q
)
-
(
u

(
H

(
x
p
,
C
p
,
C
r
)
·
m
p
,
q
,
r
(
p
)
)
-
u

(
H

(
x
r
,
C
p
,
C
r
)
·
m
p
,
q
,
r
(
r
)
)
)

1
d

(
C
p
-
C
r
)



with



x

=
∑
l

x
l
2
wherein argmin denotes the minimization criteria and p, q and r denote indices of individual cameras. Cp, Cq, Cr denote camera positions of cameras p, q and r, d(Cp−Cq) is a distance between the camera positions, xp, xq and xr are vectors with parameters to be determined so as to minimizing the error for camera p, q and r, H(⋅) is a function providing a homography matrix undistorting points according to a parameter vector xp and camera positions Cp and Cq, mp,q=[m(p)p,q m(q)p,q] denotes a set of reference points wherein m(p)p,q corresponds to the respective reference points in camera p, mp,q,r=[m(p)p,q,r m(q)p,q,r m(r)p,q,r] denotes a set of reference points wherein m(p)p,q,r corresponds to the respective reference points in camera p, Functions u(⋅) and v(⋅) extract the horizontal/vertical component of an image point respectively, the set Ω comprises all pairs of partially overlapping images;
the set ψ comprises triplets of partially overlapping images;
such that each of the N cameras considered in the minimization problem is comprised in at least one pair of cameras comprised in Ω and at least in one triplet of cameras comprised in ψ;
and/or
determining the displacement information minimizing an error of a linearized first minimization criteria, wherein for a 2D camera pattern the linearized first minimization criteria is based on the determination rule;

argmin
x
1
,
x
2
,
x
3
,
…
,
x
N

∑
(
p
,
q
)
∈
Ω

[
J
p
,
q
(
p
)

J
p
,
q
(
q
)
]
·
[
x
p
T
x
q
T
]
+
b
p
,
q

wherein


(
v

(
H

(
x
p
,
C
p
,
C
q
)
·
m
p
,
q
(
p
)
)
-
v

(
H

(
x
q
,
C
p
,
C
q
)
·
m
p
,
q
(
q
)
)
)
2
≈
[
J
p
,
q
(
p
)

J
p
,
q
(
q
)
]
·
[
x
p
T
x
q
T
]
+
b
p
,
q
determining the offset information minimizing an error of a linearized second minimization criteria, wherein for the 2D camera pattern the linearized second minimization criteria is based on the determination rule

argmin
x
1
,
x
2
,
x
3
,
…
,
x
N

∑
(
p
,
q
,
r
)
∈
Ψ

[
J
p
,
q
,
r
(
p
)


J
p
,
q
,
r
(
q
)


J
p
,
q
,
r
(
r
)
]
·
[
x
p
T
x
q
T
x
r
T
]
+
b
p
,
q
,
r

wherein


(
(
u

(
H

(
x
p
,
C
p
,
C
q
)
·
m
p
,
q
,
r
(
p
)
)
-
u

(
H

(
x
q
,
C
p
,
C
q
)
·
m
p
,
q
,
r
(
q
)
)
)

1
d

(
C
p
-
C
q
)
-
(
u

(
H

(
x
p
,
C
p
,
C
r
)
·
m
p
,
q
,
r
(
p
)
)
-
u

(
H

(
x
r
,
C
p
,
C
r
)
·
m
p
,
q
,
r
(
r
)
)
)

1
d

(
C
p
-
C
r
)
)
2
≈
[
J
p
,
q
,
r
(
p
)


J
p
,
q
,
r
(
q
)


J
p
,
q
,
r
(
r
)
]
·
[
x
p
T
x
q
T
x
r
T
]
+
b
p
,
q
,
r
wherein argmin denotes the minimization criteria p, q and r denote indices of individual cameras, Cp, Cq, Cr denote camera positions of cameras p, q and r, d(Cp−Cq) is a distance between the camera positions, H(⋅) is a function providing a homography matrix undistorting points according to a parameter vector xp and camera positions Cp and Cq, corresponding to camera p with parameter vector xp, xq and xr, are vectors with parameters to be determined so as to minimizing the error for camera p, q and r, mp,q=[m(p)p,q m(q)p,q] denotes a set of reference points wherein m(p)p,q corresponds to the respective reference points in camera p, mp,q,r=[m(p)p,q,r m(q)p,q,r m(r)p,q,r] denotes a set of reference points wherein m(p)p,q,r corresponds to the respective reference points in camera p, the set Ω comprises all pairs of partially overlapping images;
the set ψ comprises triplets of partially overlapping images;
such that each of the N cameras considered in the minimization problem is comprised in at least one pair of cameras comprised in Ω and in at least one triplet of cameras comprised in ψ, functions u(⋅) and v(⋅) extract the horizontal/vertical component of an image point respectively, Jp,q=[J(p)p,q J(q)p,q] and Jp,q,r=[J(p)p,q,r J(q)p,q,r J(r)p,q,r] denote the Jacobi matrices for camera pairs and camera triplets, respectively, J(p)p,q and J(p)p,q,r denote sub-matrices corresponding to a camera p, bp,q and bp,q,r denote residual elements.

698 more elements

A non-transitory digital storage medium having a computer program stored thereon to perform the method for acquiring calibration data, the method comprising:

receiving a plurality of partially overlapping images of an object from a corresponding plurality of camera positions being arranged along a first and a second direction according to a two-dimensional or three-dimensional camera pattern, wherein patterns comprising camera positions that differ in two directions are two-dimensional patterns and patterns comprising camera positions that differ in three directions are three-dimensional patterns;
selecting at least one corresponding reference point in an overlap area of a set of overlapping images, and determining a displacement information along the first and the second direction of the reference point in each of the other images of the set of images, such that a misalignment of the plurality of images along the first and the second direction is compensated by the displacement information so as to acquire aligned images;
determining an offset information between principal points at the plurality of camera positions using at least three aligned images;
and
providing calibration data based on the displacement information and based on the offset information, the calibration data allowing for calibrating the plurality of images so as to comply to the camera pattern;
and
determining the displacement information minimizing an error of a first minimization criteria, wherein for a 2D camera pattern the first minimization criteria is based on the determination rule;
argmin
x
1
,
x
2
,
x
3
,
…
,
x
N

∑
(
p
,
q
)
∈
Ω


v

(
H

(
x
p
,
C
p
,
C
q
)
·
m
p
,
q
(
p
)
)
-
v

(
H

(
x
q
,
C
p
,
C
q
)
·
m
p
,
q
(
q
)
)




with




x


=
∑
l

x
l
2
determining the offset information minimizing an error of a second minimization criteria, wherein for the 2D camera pattern the second minimization criteria is based on the determination rule
argmin
x
1
,
x
2
,
x
3
,
…
,
x
N

∑
(
p
,
q
,
r
)
∈
Ψ


(
u

(
H

(
x
p
,
C
p
,
C
q
)
·
m
p
,
q
,
r
(
p
)
)
-
u

(
H

(
x
q
,
C
p
,
C
q
)
·
m
p
,
q
,
r
(
q
)
)
)

1
d

(
C
p
-
C
q
)
-
(
u

(
H

(
x
p
,
C
p
,
C
r
)
·
m
p
,
q
,
r
(
p
)
)
-
u

(
H

(
x
r
,
C
p
,
C
r
)
·
m
p
,
q
,
r
(
r
)
)
)

1
d

(
C
p
-
C
r
)



with



x

=
∑
l

x
l
2
wherein argmin denotes the minimization criteria and p, q and r denote indices of individual cameras. Cp, Cq, Cr denote camera positions of cameras p, q and r, d(Cp−Cq) is a distance between the camera positions, xp, xq and xr are vectors with parameters to be determined so as to minimizing the error for camera p, q and r, H(⋅) is a function providing a homography matrix undistorting points according to a parameter vector xp and camera positions Cp and Cq, mp,q=[m(p)p,q m(q)p,q] denotes a set of reference points wherein m(p)p,q corresponds to the respective reference points in camera p, mp,q,r=[m(p)p,q,r m(q)p,q,r m(r)p,q,r] denotes a set of reference points wherein m(p)p,q,r corresponds to the respective reference points in camera p, Functions u(⋅) and v(⋅) extract the horizontal/vertical component of an image point respectively, the set Ω comprises all pairs of partially overlapping images;
the set ψ comprises triplets of partially overlapping images;
such that each of the N cameras considered in the minimization problem is comprised in at least one pair of cameras comprised in Ω and at least in one triplet of cameras comprised in ψ;
and/or
determining the displacement information minimizing an error of a linearized first minimization criteria, wherein for a 2D camera pattern the linearized first minimization criteria is based on the determination rule;

argmin
x
1
,
x
2
,
x
3
,
…
,
x
N

∑
(
p
,
q
)
∈
Ω

[
J
p
,
q
(
p
)

J
p
,
q
(
q
)
]
·
[
x
p
T
x
q
T
]
+
b
p
,
q

wherein
(
v

(
H

(
x
p
,
C
p
,
C
q
)
·
m
p
,
q
(
p
)
)
-
v

(
H

(
x
q
,
C
p
,
C
q
)
·
m
p
,
q
(
q
)
)
)
2
≈
[
J
p
,
q
(
p
)

J
p
,
q
(
q
)
]
·
[
x
p
T
x
q
T
]
+
b
p
,
q
determining the offset information minimizing an error of a linearized second minimization criteria, wherein for the 2D camera pattern the linearized second minimization criteria is based on the determination rule

argmin
x
1
,
x
2
,
x
3
,
…
,
x
N

∑
(
p
,
q
,
r
)
∈
Ψ

[
J
p
,
q
,
r
(
p
)


J
p
,
q
,
r
(
q
)


J
p
,
q
,
r
(
r
)
]
·
[
x
p
T
x
q
T
x
r
T
]
+
b
p
,
q
,
r

wherein


(
(
u

(
H

(
x
p
,
C
p
,
C
q
)
·
m
p
,
q
,
r
(
p
)
)
-
u

(
H

(
x
q
,
C
p
,
C
q
)
·
m
p
,
q
,
r
(
q
)
)
)

1
d

(
C
p
-
C
q
)
-
(
u

(
H

(
x
p
,
C
p
,
C
r
)
·
m
p
,
q
,
r
(
p
)
)
-
u

(
H

(
x
r
,
C
p
,
C
r
)
·
m
p
,
q
,
r
(
r
)
)
)

1
d

(
C
p
-
C
r
)
)
2
≈
[
J
p
,
q
,
r
(
p
)


J
p
,
q
,
r
(
q
)


J
p
,
q
,
r
(
r
)
]
·
[
x
p
T
x
q
T
x
r
T
]
+
b
p
,
q
,
r
wherein argmin denotes the minimization criteria p, q and r denote indices of individual cameras, Cp, Cq, Cr denote camera positions of cameras p, q and r, d(Cp−Cq) is a distance between the camera positions, H(⋅) is a function providing a homography matrix undistorting points according to a parameter vector xp and camera positions Cp and Cq, corresponding to camera p with parameter vector xp, xq and xr, are vectors with parameters to be determined so as to minimizing the error for camera p, q and r, mp,q=[m(p)p,q m(q)p,q] denotes a set of reference points wherein m(p)p,q corresponds to the respective reference points in camera p, mp,q,r=[m(p)p,q,r m(q)p,q,r m(r)p,q,r] denotes a set of reference points wherein m(p)p,q,r corresponds to the respective reference points in camera p, the set Ω comprises all pairs of partially overlapping images;
the set ψ comprises triplets of partially overlapping images;
such that each of the N cameras considered in the minimization problem is comprised in at least one pair of cameras comprised in Ω and in at least one triplet of cameras comprised in ψ, functions u(⋅) and v(⋅) extract the horizontal/vertical component of an image point respectively, Jp,q=[J(p)p,q J(q)p,q] and Jp,q,r=[J(p)p,q,r J(q)p,q,r J(r)p,q,r] denote the Jacobi matrices for camera pairs and camera triplets, respectively, J(p)p,q and J(p)p,q,r denote sub-matrices corresponding to a camera p, bp,q and bp,q, denote residual elements,
when said computer program is run by a computer.

698 more elements

Apparatus for providing calibration data, camera system and method for obtaining calibration data

Life of this patent

Apparatus for providing calibration data, camera system and method for obtaining calibration data

Life of this patent

Abstract

Claims18