# US 2021/0343044 A1 · Vehicle camera calibration apparatus and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0343044 A1
- Document ID: US20210343044A1
- Status: Published application
- Kind: A1
- Application number: 16/315,941
- Filed: 2018-08-08
- Published: 2021-11-04
- Earliest claimed: 2017-09-13
- Assignees: VADAS Co., Ltd.
- Applicants: VADAS Co., Ltd.
- Inventors: Jun-Seok LEE; Byeong-Chan JEON; Jeong-mok HA; Hee-soo KIM
- Web page: https://patentgrep.com/patent/US20210343044A1

## CPC

- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- B60R 1/27: providing all-round vision, e.g. using omnidirectional cameras
- G06T 5/70: Denoising; Smoothing
- G06T 7/73: using feature-based methods
- G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road}
- H04N 5/76: Television signal recording
- H04N 17/002: {for television cameras}
- H04N 23/695: Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- B60R 2300/105: using multiple cameras
- B60R 2300/303: using joined images, e.g. multiple camera images
- B60R 2300/307: virtually distinguishing relevant parts of a scene from the background of the scene
- B60R 2300/402: Image calibration
- B60R 2300/607: from a bird's eye viewpoint
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T 2207/30256: Lane; Road marking
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A vehicle camera calibration apparatus and method are provided. The vehicle camera calibration apparatus includes a camera module configured to acquire an image representing a road from a plurality of cameras installed in a vehicle, an input/output module configured to receive, as an input, the acquired image from the camera module, or output a corrected image, a lane detection module configured to detect a lane and extract a feature point of the lane from an image received from the input/output module, and a camera correction module configured to estimate a new external parameter using a lane equation and a lane width based on initial camera information and external parameter information in the image received from the input/output module, and to correct the image.

## Claims

### Claim 1

A vehicle camera calibration apparatus comprising:
a camera module configured to acquire an image representing a road from a plurality of cameras installed in a vehicle;
an input/output module configured to receive, as an input, the acquired image from the camera module, or output a corrected image;
a lane detection module configured to detect a lane and extract a feature point of the lane from an image received from the input/output module; and
a camera correction module configured to calculate a lane equation and a lane width based on initial camera information and external parameter information in the image received from the input/output module, to estimate a new external parameter, and to correct the image.

### Claim 2 (depends on 1)

The vehicle camera calibration apparatus of claim 1, wherein the input/output module comprises:
an image inputter/outputter configured to receive, as an input, an image from the camera module, or to output a corrected image to another module; and
a storage configured to receive the corrected image and changed external parameter information from the camera correction module, and to store the corrected image and the changed external parameter information.

### Claim 3 (depends on 1)

The vehicle camera calibration apparatus of claim 1, wherein the lane detection module comprises:
an image processor configured to remove noise from the image received from the input/output module;
a frame accumulator configured to accumulate images processed by the image processor as consecutive frames, when a lane is marked with a dashed line;
an edge detector configured to detect an edge and to extract a feature point from a cumulative image including the images accumulated by the frame accumulator; and
a lane determiner configured to determine a lane from the feature point extracted by the edge detector.

### Claim 4 (depends on 1)

The vehicle camera calibration apparatus of claim 1, wherein the camera correction module comprises:
an image converter configured to extract a feature point of a lane from a received image based on initial camera information and external parameter information, and to convert the received image into a top-view image;
an equation calculator configured to acquire a lane equation from the top-view image acquired by the image converter;
a parameter estimator configured to estimate a new external parameter using the lane equation calculated by the equation calculator; and
a corrector configured to correct an image using the new external parameter estimated by the parameter estimator.

### Claim 5

A vehicle camera calibration method comprising:
a lane detection operation of receiving, as an input, an acquired image from a camera module, and detecting a lane;
an image conversion operation of extracting a feature point of a lane from a received image based on initial camera information and external parameter information, and converting the received image into a top-view image;
an equation calculation operation of acquiring a lane equation and a lane width from the top-view image acquired in the image conversion operation;
a parameter estimation operation of estimating a new external parameter using the lane equation and the lane width calculated in the equation calculation operation; and
an image correction operation of correcting an image using the new external parameter estimated in the parameter estimation operation.

### Claim 6 (depends on 5)

The vehicle camera calibration method of claim 5, wherein the lane detection operation comprises:
an image processing operation of removing noise from an image received from an input/output module;
a frame accumulation operation of accumulating images processed in the image processing operation as consecutive frames when a lane is marked with a dashed line in an image processed in the image processing operation, and performing a next operation when the lane is marked with a straight line;
an edge detection operation of detecting an edge and extracting a feature point from a cumulative image including the images accumulated in the frame accumulation operation; and
a lane determination operation of determining a lane from the feature point extracted in the edge detection operation.

### Claim 7 (depends on 6)

The vehicle camera calibration method of claim 6, wherein the image processing operation comprises performing an algorithm of softening a boundary of an object in an image to remove noise.

### Claim 8 (depends on 6)

The vehicle camera calibration method of claim 6, wherein the image processing operation comprises removing noise by changing a size of an image to prevent a region corresponding to a split lane in an acquired image from being detected as an edge.

### Claim 9 (depends on 6)

The vehicle camera calibration method of claim 6, wherein the frame accumulation operation comprises comparing all pixel values of the cumulative image and storing a highest pixel value in a new image so that a dashed-line lane in an acquired image is viewed to be identical to a solid-line lane.

### Claim 10 (depends on 6)

The vehicle camera calibration method of claim 6, wherein the edge detection operation comprises detecting an edge using an algorithm and extracting the edge and central coordinates of the edge as feature points.

### Claim 11 (depends on 6)

The vehicle camera calibration method of claim 6, wherein the lane determination operation comprises detecting a longest outline among lines represented by feature points extracted in the edge detection operation and determining the longest outline as a lane.

### Claim 12 (depends on 5)

The vehicle camera calibration method of claim 5, wherein the lane equation calculated in the equation calculation operation is expressed as an equation of a straight line, or a multi-dimensional equation.

### Claim 13 (depends on 5)

The vehicle camera calibration method of claim 5, wherein the parameter estimation operation comprises:
a primary correction operation of determining whether correction information of a pitch (Rx), a yaw (Ry) and a roll (Rz) of a front camera of a vehicle, and correction information of a pitch (Rx), a yaw (Ry) and a roll (Rz) of a rear camera of the vehicle are accurate, of performing a next operation when the correction information is accurate, and of calculating the correction information of the pitch (Rx), the yaw (Ry) and the roll (Rz) of the front camera and the correction information of the pitch (Rx), the yaw (Ry) and the roll (Rz) of the rear camera and returning to the image conversion operation when the correction information is inaccurate;
a secondary correction operation of determining whether correction information of a position translation (Tz) of a rear camera of a vehicle is accurate, of performing a next operation when the correction information is accurate, and of calculating the correction information of the position translation (Tz) of the rear camera and returning to the image conversion operation when the correction information is inaccurate;
a tertiary correction operation of determining whether correction information of a pitch (Rx), a yaw (Ry), a roll (Rz) and a position translation (Tz) of each of a left camera and a right camera of a vehicle is accurate, of performing a next operation when the correction information is accurate, and of calculating the correction information of the pitch (Rx), the yaw (Ry), the roll (Rz) and the position translation (Tz) of each of the left camera and the right camera and returning to the image conversion operation when the correction information is inaccurate; and
a storage operation of storing external parameters calculated in the primary correction operation through the tertiary correction operation.

### Claim 14 (depends on 13)

The vehicle camera calibration method of claim 13, wherein the correction information of the pitch (Rx) of each of the front camera and the rear camera in the primary correction operation is calculated by modifying a new pitch (Rx) and repeating a process from the image conversion operation until Equation 1 shown below is satisfied.
1
G
1
-
1
G
r
=
0
[
Equation
⁢
⁢
1
]
wherein Gl denotes a slope of a left lane, and Gr denotes a slope of a right lane.

### Claim 15 (depends on 13)

The vehicle camera calibration method of claim 13, wherein the correction information of the yaw (Ry) of each of the front camera and the rear camera in the primary correction operation and the secondary correction operation is calculated by modifying a new yaw (Ry) and repeating a process from the image conversion operation until Equation 2 shown below is satisfied.
W1−Wr=0 [Equation 2]
wherein Wl denotes a width of a left lane, and Wr denotes a width of a right lane.

### Claim 16 (depends on 13)

The vehicle camera calibration method of claim 13, wherein the correction information of the roll (Rz) of each of the front camera and the rear camera in the primary correction operation is calculated by modifying a new roll (Rz) and repeating a process from the image conversion operation until Equation 3 shown below is satisfied.
1
G
1
-
1
G
r
=
0
[
Equation
⁢
⁢
3
]
wherein Gl denotes a slope of a left lane, and Gr denotes a slope of a right lane.

### Claim 17 (depends on 13)

The vehicle camera calibration method of claim 13, wherein the correction information of the position translation (Tz) of the rear camera in the secondary correction operation is calculated by modifying a new position translation (Tz) and repeating a process from the image conversion operation until Equation 4 shown below is satisfied.
Cf−Cr=0 [Equation 4]
wherein Cf denotes a width between a left lane and a right lane in a front road, and Cr denotes a width between a left lane and a right lane in a rear road.

### Claim 18 (depends on 13)

The vehicle camera calibration method of claim 13, wherein
the correction information of the pitch (Rx) of the left camera in the tertiary correction operation is calculated by modifying a new pitch (Rx) and repeating a process from the image conversion operation until Equation 5 shown below is satisfied, and
the correction information of the pitch (Rx) of the right camera in the tertiary correction operation is calculated by modifying a new pitch (Rx) and repeating a process from the image conversion operation until Equation 6 shown below is satisfied.
(
x
2
+
x
6
)
2
-
(
x
1
+
x
5
)
2
=
0
[
Equation
⁢
⁢
5
]
(
x
3
+
x
7
)
2
-
(
x
4
+
x
8
)
2
=
0
[
Equation
⁢
⁢
6
]
wherein x1 denotes a left x-coordinate of a left lane of a front road, x2 denotes a right x-coordinate of the left lane of the front road, x5 denotes a left x-coordinate of a left lane of a rear road, x6 denotes a right x-coordinate of the left lane of the rear road, x3 denotes a left x-coordinate of a right lane of the front road, x4 denotes a right x-coordinate of the right lane of the front road, x7 denotes a left x-coordinate of a right lane of the rear road, and x8 denotes a right x-coordinate of the right lane of the rear road.

### Claim 19 (depends on 13)

The vehicle camera calibration method of claim 13, wherein
the correction information of the yaw (Ry) of the left camera in the tertiary correction operation is calculated by modifying a new yaw (Ry) and repeating a process from the image conversion operation until Equation 7 shown below is satisfied, and
the correction information of the yaw (Ry) of the right camera in the tertiary correction operation is calculated by modifying a new yaw (Ry) and repeating a process from the image conversion operation until Equation 8 shown below is satisfied.
(x22−x11)−(x66−x55)=0[Equation 7]
(x44−x33)−(x88−x77)=0 [Equation 8]
wherein x11 denotes a left x-coordinate of a left lane of a front road, x22 denotes a right x-coordinate of the left lane of the front road, x55 denotes a left x-coordinate of a left lane of a rear road, x66 denotes a right x-coordinate of the left lane of the rear road, x33 denotes a left x-coordinate of a right lane of the front road, x44 denotes a right x-coordinate of the right lane of the front road, x77 denotes a left x-coordinate of a right lane of the rear road, and x88 denotes a right x-coordinate of the right lane of the rear road.

### Claim 20 (depends on 13)

The vehicle camera calibration method of claim 13, wherein
the correction information of the roll (Rz) of the left camera in the tertiary correction operation is calculated by modifying a new roll (Rz) and repeating a process from the image conversion operation until Equation 9 shown below is satisfied, and
the correction information of the roll (Rz) of the right camera in the tertiary correction operation is calculated by modifying a new roll (Rz) and repeating a process from the image conversion operation until Equation 10 shown below is satisfied.
1
G
1
=
0
[
Equation
⁢
⁢
9
]
1
G
r
=
0
[
Equation
⁢
⁢
10
]
wherein Gl denotes a slope of a left lane, and Gr denotes a slope of a right lane.

### Claim 21 (depends on 13)

The vehicle camera calibration method of claim 13, wherein
the correction information of the position translation (Tz) of the left camera in the tertiary correction operation is calculated by modifying a new position translation (Tz) and repeating a process from the image conversion operation until Equation 11 shown below is satisfied, and
the correction information of the position translation (Tz) of the right camera in the tertiary correction operation is calculated by modifying a new position translation (Tz) and repeating a process from the image conversion operation until Equation 12 shown below is satisfied.
(
x
2
⁢
2
⁢
2
-
x
111
)
+
(
x
10
-
x
999
)
2
-
(
x
666
-
x
5
⁢
5
⁢
5
)
=
0
[
Equation
⁢
⁢
11
]
(
x
4
⁢
4
⁢
4
-
x
3
⁢
3
⁢
3
)
+
(
x
1
⁢
2
-
x
1
⁢
1
)
2
-
(
x
888
-
x
777
)
=
0
[
Equation
⁢
⁢
12
]
wherein x111 denotes a left x-coordinate of a left lane of a front road, x222 denotes a right x-coordinate of the left lane of the front road, x555 denotes a left x-coordinate of a left lane, x666 denotes a right x-coordinate of the left lane, x999 denotes a left x-coordinate of a left lane of the rear road, x10 denotes a right x-coordinate of the left lane of the rear road, x333 denotes a left x-coordinate of a right lane of a front road, x444 denotes a right x-coordinate of a right lane of a front road, x777 denotes a left x-coordinate of a right lane, x888 denotes a right x-coordinate of a right lane, x11 denotes a left x-coordinate of a right lane of the rear road, and x12 denotes a right x-coordinate of the right lane of the rear road.
