# US 10,373,338 B2 · Calculation device, camera device, vehicle, and calibration method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,373,338 B2
- Document ID: US10373338B2
- Status: Granted
- Kind: B2
- Application number: 15/576,781
- Filed: 2016-05-26
- Published: 2018-06-14
- Granted: 2019-08-06
- Earliest claimed: 2016-05-26
- Assignees: KYOCERA Corporation
- Applicants: KYOCERA Corporation
- Inventors: Shushin Inoue; Tomofumi Koishi
- Web page: https://patentgrep.com/patent/US10373338B2

## CPC

- G06T 7/85: {Stereo camera calibration}
- G01C 3/14: with binocular observation at a single point, e.g. stereoscopic type (G01C3/20 takes precedence)
- G01C 21/28: with correlation of data from several navigational instruments
- G01C 25/00: Manufacturing, calibrating, cleaning, or repairing instruments or devices referred to in the other groups of this subclass (testing, calibrating or compensating compasses G01C17/38)
- G06T 3/00: Geometric image transformations in the plane of the image
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 13/246: Calibration of cameras
- H04N 17/002: {for television cameras}
- H04N 23/60: Control of cameras or camera modules
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G06T 2207/30256: Lane; Road marking
- G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road}
- G08G 1/165: {for passive traffic, e.g. including static obstacles, trees}
- G08G 1/166: {for active traffic, e.g. moving vehicles, pedestrians, bikes}
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})

## Abstract

A calculation device comprises a controller configured to extract, from a captured image of a subject, a plurality of lines corresponding to straight lines parallel to each other in the subject, and to calculate first calibration data so that mapping lines obtained when the extracted plurality of lines are mapped to a three-dimensional coordinate space are parallel to each other.

## Claims

### Claim 1

A calculation device comprising:
a controller configured to extract, from a captured image of a subject, a plurality of lines corresponding to straight lines parallel to each other located on a plane in the subject, to map the extracted plurality of lines to a three-dimensional coordinate space, and to calculate first calibration data so that mapping lines obtained are parallel to each other,
wherein the controller is configured to calibrate an angle θC between an optical axis of a camera that captures the image and the plane, based on the first calibration data, and
wherein the mapping of the plurality of lines to the three-dimensional coordinate space is conducted according to the following conversion expressions, when coordinate on the captured image expressed as a pixel value is (u, v), and coordinate on XZ-plane in the three-dimensional coordinate space corresponding to the plane is (X, Z),
Z
=
H
C
·
f
-
(
v
-
v
0
)
⁢
p
·
tan
⁢
⁢
θ
C
f
·
tan
⁢
⁢
θ
C
-
(
v
-
v
0
)
⁢
p
X
=
Z
·
(
u
-
u
0
)
·
p
f
where (u0, v0) is a point on the optical axis of the camera in the captured image, HC is height of the camera from the plane, f is focal length of lens of the camera, and p is pixel pitch of the camera.

### Claim 2 (depends on 1)

The calculation device according to claim 1,
wherein the controller is configured to:
calculate the first calibration data for each of a plurality of captured images of the same subject, the plurality of images having parallax therebetween; and
calculate second calibration data by comparing corresponding mapping lines in the plurality of images.

### Claim 3 (depends on 2)

The calculation device according to claim 2,
wherein the controller is configured to perform calibration for a parallax deviation between a plurality of cameras that captures the plurality of images, based on the second calibration data.

### Claim 4

A camera device comprising:
a calculation device including a controller configured to extract, from a captured image of a subject, a plurality of lines corresponding to straight lines parallel to each other located on a plane in the subject, to map the extracted plurality of lines to a three-dimensional coordinate space, and to calculate first calibration data so that mapping lines obtained are parallel to each other; and
one or more cameras configured to capture the image of the subject,
wherein the controller is configured to calibrate an angle θC between an optical axis of at least one camera of the one or more cameras that captures the image and the plane, based on the first calibration data, and
wherein the mapping of the plurality of lines to the three-dimensional coordinate space is conducted according to the following conversion expressions, when coordinate on the captured image expressed as a pixel value is (u, v), and coordinate on XZ-plane in the three-dimensional coordinate space corresponding to the plane is (X, Z),
Z
=
H
C
·
f
-
(
v
-
v
0
)
⁢
p
·
tan
⁢
⁢
θ
C
f
·
tan
⁢
⁢
θ
C
-
(
v
-
v
0
)
⁢
p
X
=
Z
·
(
u
-
u
0
)
·
p
f
where (u0, v0) is a point on the optical axis of the at least one camera in the captured image, HC is height of the at least one camera from the plane, f is focal length of lens of the at least one camera, and p is pixel pitch of the at least one camera.

### Claim 5

A vehicle comprising:
a calculation device including a controller configured to extract, from a captured image of a subject, a plurality of lines corresponding to straight lines parallel to each other located on a plane in the subject, to map the extracted plurality of lines to a three-dimensional coordinate space, and to calculate first calibration data so that mapping lines obtained are parallel to each other; and
one or more cameras configured to capture the image of the subject,
wherein the controller is configured to calibrate an angle θC between an optical axis of at least one camera of the one or more cameras that captures the image and the plane, based on the first calibration data, and
wherein the mapping of the plurality of lines to the three-dimensional coordinate space is conducted according to the following conversion expressions, when coordinate on the captured image expressed as a pixel value is (u, v), and coordinate on XZ-plane in the three-dimensional coordinate space corresponding to the plane is (X, Z),
Z
=
H
C
·
f
-
(
v
-
v
0
)
⁢
p
·
tan
⁢
⁢
θ
C
f
·
tan
⁢
⁢
θ
C
-
(
v
-
v
0
)
⁢
p
X
=
Z
·
(
u
-
u
0
)
·
p
f
where (u0, v0) is a point on the optical axis of the at least one camera in the captured image, HC is height of the at least one camera from the plane, f is focal length of lens of the at least one camera, and p is pixel pitch of the at least one camera.

### Claim 6 (depends on 5)

The vehicle according to claim 5,
wherein the one or more cameras face outside the vehicle.

### Claim 7 (depends on 5)

The vehicle according to claim 5,
wherein the controller is configured to perform calibration based on the first calibration data, using a mounting angle of depression of each of the cameras as a reference value.

### Claim 8 (depends on 5)

The vehicle according to claim 5,
wherein the controller is configured to extract the plurality of straight lines from a ground portion in the image.

### Claim 9

A calibration method comprising:
extracting, from a captured image of a subject, a plurality of lines corresponding to straight lines parallel to each other located on a plane in the subject;
mapping the extracted plurality of lines to a three-dimensional coordinate space; and
calculating first calibration data so that mapping lines obtained are parallel to each other,
wherein an angle θC between an optical axis of a camera that captures the image and the plane is calibrated, based on the first calibration data, and
wherein the mapping of the plurality of lines to the three-dimensional coordinate space is conducted according to the following conversion expressions, when coordinate on the captured image expressed as a pixel value is (u, v), and coordinate on XZ-plane in the three-dimensional coordinate space corresponding to the plane is (X, Z),
Z
=
H
C
·
f
-
(
v
-
v
0
)
⁢
p
·
tan
⁢
⁢
θ
C
f
·
tan
⁢
⁢
θ
C
-
(
v
-
v
0
)
⁢
p
X
=
Z
·
(
u
-
u
0
)
·
p
f
where (u0, v0) is a point on the optical axis of the camera in the captured image, HC is height of the camera from the plane, f is focal length of lens of the camera, and p is pixel pitch of the camera.

### Claim 10 (depends on 9)

The calibration method according to claim 9, comprising:
calculating the first calibration data for each of a plurality of images having parallax therebetween; and
calculating second calibration data by comparing corresponding mapping lines in the plurality of images.