# US 10,482,617 B2 · Distance estimation method based on handheld light field camera

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,482,617 B2
- Document ID: US10482617B2
- Status: Granted
- Kind: B2
- Application number: 16/212,344
- Filed: 2018-12-06
- Published: 2019-04-18
- Granted: 2019-11-19
- Earliest claimed: 2016-09-18
- Assignees: Graduate School At Shenzhen, Tsinghua University
- Applicants: Graduate School at Shenzhen, Tsinghua University
- Inventors: Xin Jin; Yanqin Chen; Qionghai Dai
- Web page: https://patentgrep.com/patent/US10482617B2

## CPC

- G06T 7/557: from light fields, e.g. from plenoptic cameras
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- H04N 23/957: Light-field or plenoptic cameras or camera modules
- G06T 2207/10052: Images from lightfield camera
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A distance estimation method based on a handheld light field camera is disclosed and includes: S1: extracting parameters of the light field camera; S2: setting a reference plane and a calibration point; S3: refocusing a collected light field image on the reference plane, to obtain a distance between a main lens and a microlens array of the light field camera, and recording an imaging diameter of the calibration point on the refocused image; and S4: inputting the parameters of the light field camera, the distance between the main lens and the microlens array, and the imaging diameter of the calibration point on the refocused image to a light propagation mathematical model, and outputting a distance of the calibration point. The present application has high efficiency and relatively high accuracy.

## Claims

### Claim 1

A distance estimation method based on a handheld light field camera, comprising the following steps:
S1: extracting parameters of alight field camera, comprising a focal length, a curvature radius, a pupil diameter, and a central thickness of a main lens of the light field camera, and a focal length of a microlens array of the light field camera;
S2: setting a reference plane and a calibration point, wherein the calibration point is set on an object whose distance needs to be estimated, and obtaining a distance between the reference plane and the main lens;
S3: refocusing a collected light field image on the reference plane, to obtain a distance between the main lens and the microlens array of the light field camera, and recording an imaging diameter of the calibration point on the refocused image; and
S4: inputting the parameters of the light field camera that are extracted in step S1, the distance between the main lens and the microlens array that is obtained in step S3, and the imaging diameter of the calibration point on the refocused image that is recorded in step S3 to a light propagation mathematical model, and outputting a distance of the calibration point.

### Claim 2 (depends on 1)

The distance estimation method according to claim 1, wherein the reference plane and the calibration point set in step S2 do not obstruct each other, and the calibration point does not overlap when being imaged.

### Claim 3 (depends on 1)

The distance estimation method according to claim 1, wherein step S3 specifically comprises refocusing the collected light field image on the reference plane by using the following formula:
L
z
⁡
[
y
j
]
=
∑
i
=
-
c
c
⁢
⁢
L
⁡
[
v
m
-
1
-
c
+
i
,
y
j
+
a
⁡
(
c
-
i
)
]
,
a
∈
R
+
,
(
1
)
wherein L denotes the light field image, Lz denotes the refocused image on the reference plane z, a denotes a specific value corresponding to the reference plane z; y={x, y} denotes position information of the light field image, v={u,v} denotes direction information of the light field image, the subscript m denotes the number of pixels of each microlens in a one-dimensional direction, c=(m−1)/2, i is an integer with a value range of [−c,c], and the subscript j denotes coordinates of the microlens in a vertical direction.

### Claim 4 (depends on 3)

The distance estimation method according to claim 3, wherein the step of obtaining a distance between the main lens and the microlens array in step S3 specifically comprises:
using a ray tracing method to obtain a coordinate calculation formula of intersections on a plane Fu:
Fi=mi×f (2),
wherein the distance between the intersections on the plane Fu is the baseline of the light field camera, f is the focal length of the main lens, and mi is the slope of a light ray between a sensor and the main lens of the light field camera;
a calculation formula of the slope ki of a light ray emitted from the object in an object space is:
k
i
=
y
j
′
-
F
i
d
out
-
f
,
(
3
)
wherein yj′ denotes a vertical coordinate of the object on the reference plane, and dout denotes the distance between the reference plane and the main lens; and
obtaining, through calculation according to the formula (3), an incident position and an emergent position (p′,q′) of the main lens that are reached by the light ray emitted from the object, and obtaining, through calculation according to the emergent position (p′,q′), the distance din between the main lens and the microlens array:
d
in
=
q
′
-
y
j
+
m
i
⁢
p
′
m
i
,
(
4
)
wherein yj denotes a vertical coordinate of the center of the microlens whose subscript is j.

### Claim 5 (depends on 1)

The distance estimation method according to claim 1, wherein the light propagation mathematical model in step S4 specifically comprises a light propagation incident part and a light propagation emergent part.

### Claim 6 (depends on 5)

The distance estimation method according to claim 5, wherein a propagation mathematical model of the light propagation incident part specifically comprises:
the light ray emitted from the calibration point enters the main lens at an angle φ, wherein φ meets the relational expression:
tan
⁢
⁢
φ
=
D
2
⁢
(
d
out
′
-
T
⁢
/
⁢
2
+
R
-
R
2
-
D
2
⁢
/
⁢
4
)
,
(
5
)
wherein dout′ denotes an axial distance between the calibration point and the center of the main lens, R denotes the curvature radius of the main lens, D denotes the pupil radius of the main lens, T denotes the central thickness of the main lens, and the light ray emitted from the calibration point is refracted after entering the main lens, and the following formula is met:
n1 sin ψ=sin(φ+θ1) (6),
wherein n1 denotes a refractive index of the main lens, ψ denotes a refraction angle, and θ1 meets:
sin
⁢
⁢
θ
1
=
D
2
⁢
R
.
(
7
)

### Claim 7 (depends on 6)

The distance estimation method according to claim 6, wherein a propagation mathematical model of the light propagation emergent part specifically comprises:
the light ray emitted from the calibration point reaches the emergent position (p,q) of the main lens after being refracted in the main lens, and is emergent from the emergent position (p,q), and the following formula is met:
n1 sin(θ1−ψ+θ2)=sin ω (8)
wherein ω denotes an emergent angle, and θ2 meets:
sin
⁢
⁢
θ
2
=
q
R
;
(
9
)
there are three imaging manners after the light ray emitted from the calibration point is emergent from the main lens, which are separately focusing on the back side of the sensor, between the sensor and the microlens array, and between the microlens array and the main lens, wherein
when the light ray emitted from the calibration point is focused on the back side of the sensor after being emergent from the main lens, the following formula is met:
D
1
d
≈
D
f
x
+
d
+
d
in
,
(
10
)
wherein D1 is the imaging diameter that is of the calibration point on the refocused image and that is recorded in step S3, fx is the focal length of the microlens, din is the distance that is between the main lens and the microlens array and that is obtained in step S3, d is the distance between the calibration point and the sensor on a focusing plane of an image space, and the emergent position (p,q) is on a curved surface of the main lens, and the following formulas are met:
tan
⁡
(
ω
-
θ
2
)
=
D
1
2
⁢
d
=
q
f
x
+
d
+
d
in
-
p
;
and
(
11
)
(
R
-
T
⁢
/
⁢
2
+
p
)
2
+
q
2
=
R
2
;
(
12
)
when the light ray emitted from the calibration point is focused between the sensor and the microlens array or between the microlens array and the main lens after being emergent from the main lens, the following formula is met:
D
1
d
≈
D
f
x
-
d
+
d
in
,
(
13
)
wherein D1 is the imaging diameter that is of the calibration point on the refocused image and that is recorded in step S3, fx is the focal length of the microlens, din is the distance that is between the main lens and the microlens array and that is obtained in step S3, d is the distance between the calibration point and the sensor on a focusing plane of an image space, and the emergent position (p,q) is on the curved surface of the main lens, and the following formulas are met:
tan
⁡
(
ω
-
θ
2
)
=
D
1
2
⁢
d
=
q
f
x
-
d
+
d
in
-
p
;
and
(
14
)
(
R
-
T
⁢
/
⁢
2
+
p
)
2
+
q
2
=
R
2
.
(
15
)