# US 2021/0241492 A1 · Vehicular vision system with camera calibration using calibration target > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0241492 A1 - Document ID: US20210241492A1 - Status: Published application - Kind: A1 - Application number: 17/248,681 - Filed: 2021-02-03 - Published: 2021-08-05 - Earliest claimed: 2020-02-03 - Assignees: Magna Electronics Inc. - Applicants: Magna Electronics Inc. - Inventors: Tao-I Hsu; James Turk; Alexander Velichko - Web page: https://patentgrep.com/patent/US20210241492A1 ## CPC - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - B60R 11/04: Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle - B60R 16/0231: {Circuits relating to the driving or the functioning of the vehicle (monitoring tyres B60C23/00; indicating overspeed B60K31/00; for dash boards B60K37/00, B60Q3/10; for indicating emergencies B60Q1/52; brake control systems B60T; registering or indicating the working of vehicles G07C5/00; measuring distance G01C, e.g. combinations of speed and distance G01C23/00; engine indicators G01L; measuring speed or acceleration G01P)} - G06T 7/13: Edge detection - G06T 7/73: using feature-based methods - H04N 17/002: {for television cameras} - B60R 2300/402: Image calibration - G06T 2207/30244: Camera pose - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method for calibrating a vehicular camera of a vehicular vision system includes placing a target with a first portion having a first geometric pattern and a second portion having a second geometric pattern within the field of view of the vehicular camera, and capturing image data with the camera representative of the field of view of the vehicular camera. Two edges of both portions of the target are detected, and edge pixels of the detected edges are determined. First and second vanishing points of the target in the captured image data are determined based on the determined edge pixels of the respective first and second portions of the target. Camera orientation is determined based on location of the determined first vanishing point relative to location of the determined second vanishing point. The vehicular camera is calibrated based on the determined camera orientation. ## Claims ### Claim 1 A method for calibrating a vehicular vision system, the method comprising: equipping a vehicle with the vehicular vision system, the vehicular vision system comprising at least one vehicular camera disposed at the vehicle; placing a target within a field of view of the vehicular camera, wherein a first portion of the target comprises a first geometric pattern and a second portion of the target comprises a second geometric pattern, and wherein, with the target placed within the field of view of the vehicular camera, the first portion and the second portion are within the field of view of the vehicular camera; capturing image data with the camera, the captured image data representative of the field of view of the vehicular camera; providing the captured image data to an image processor; detecting, via processing at the image processor of the captured image data, first and second edges of the first portion of the target and third and fourth edges of the second portion of the target; determining, via processing at the image processor of the captured image data, first edge pixels representative of the first detected edge of the first portion of the target, second edge pixels representative of the second detected edge of the first portion of the target, third edge pixels representative of the third detected edge of the second portion of the target, and fourth edge pixels representative of the fourth detected edge of the second portion of the target; determining a first vanishing point based on the determined first edge pixels of the first portion of the target and the determined second edge pixels of the first portion of the target, and determining a second vanishing point based on the determined third edge pixels of the second portion of the target and the determined fourth edge pixels of the second portion of the target; determining camera orientation based on location of the determined first vanishing point relative to location of the determined second vanishing point; and calibrating the vehicular vision system for the vehicular camera based on the determined camera orientation. ### Claim 2 (depends on 1) The method of claim 1, wherein detecting edges of the target comprises performing a Canny edge detection. ### Claim 3 (depends on 1) The method of claim 1, wherein detecting edges of the target comprises reducing a zig-zag effect of the captured image data. ### Claim 4 (depends on 1) The method of claim 1, wherein determining the first and second edge pixels of the first and second detected edges of the first portion of the target and determining the third and fourth edge pixels of the third and fourth detected edges of the second portion of the target comprises determining a line distance and a line angle based on a pixel gradient and coefficients. ### Claim 5 (depends on 1) The method of claim 1, wherein determining the first and second edge pixels of the first and second detected edges of the first portion of the target and determining the third and fourth edge pixels of the third and fourth detected edges of the second portion of the target comprises grouping the edge pixels of the respective detected edges into respective lines. ### Claim 6 (depends on 1) The method of claim 1, wherein determining the first and second vanishing points comprises performing a J-linkage operation to ensure pair-parallel lines cluster with minimum distance. ### Claim 7 (depends on 1) The method of claim 1, wherein determining the first and second vanishing points comprises performing a simulated annealing algorithm. ### Claim 8 (depends on 1) The method of claim 1, comprising determining, via processing by the image processor of the captured image data, at least two orthogonal vanishing point candidates based on the determined first vanishing point of the first portion of the target and based on the determined second vanishing point of the second portion of the target. ### Claim 9 (depends on 8) The method of claim 8, wherein determining two orthogonal vanishing point candidates comprises estimating location of the horizon and zenith. ### Claim 10 (depends on 1) The method of claim 1, wherein the first and second portions of the target comprise at least two different sides of the target in the field of view of the vehicular camera with each side including a geometric pattern that comprises orthogonal lines. ### Claim 11 (depends on 1) The method of claim 1, further comprising, prior to detecting the first and second edges of the first portion of the target and the third and fourth edges of the second portion of the target, dewarping the captured image data. ### Claim 12 (depends on 1) The method of claim 1, further comprising determining an orientation matrix based on the determined first and second vanishing points and determining the camera orientation based on the orientation matrix. ### Claim 13 (depends on 12) The method of claim 12, wherein determining camera orientation based on the determined orientation matrix comprises determining (i) a pitch, (ii) a yaw, and (iii) a roll of the camera. ### Claim 14 A vehicular vision calibration system, the system comprising: a camera disposed at a vehicle and having a field of view exterior of the vehicle, the camera capturing image data; an electronic control unit (ECU) comprising electronic circuitry and associated software; wherein the electronic circuitry of the ECU comprises an image processor for processing image data captured by the camera to detect presence of objects in the field of view of the camera; wherein the ECU, responsive to processing by the image processor of image data captured by the camera, captures image data representative of a target within the field of view of the camera, wherein a first portion of the target comprises a first geometric pattern and a second portion of the target comprises a second geometric pattern, and wherein, with the captured image data is representative of both the first portion and the second portion of the target; wherein the ECU detects, via processing at the image processor of the captured image data, first and second edges of the first portion of the target and third and fourth edges of the second portion of the target; wherein the ECU determines, via processing at the image processor of the captured image data, first edge pixels representative of the first detected edge of the first portion of the target, second edge pixels representative of the second detected edge of the first portion of the target, third edge pixels representative of the third detected edge of the second portion of the target, and fourth edge pixels representative of the fourth detected edge of the second portion of the target; wherein the ECU determines a first vanishing point based on the determined first edge pixels of the first portion of the target and the determined second edge pixels of the first portion of the target, and wherein the ECU determines a second vanishing point based on the determined third edge pixels of the second portion of the target and the determined fourth edge pixels of the second portion of the target; wherein the ECU determines camera orientation of the camera based on location of the determined first vanishing point relative to location of the determined second vanishing point; and wherein the ECU calibrates the camera based on the determined camera orientation. ### Claim 15 (depends on 14) The system of claim 14, wherein the ECU detects edges of the target by performing a Canny edge detection. ### Claim 16 (depends on 14) The system of claim 14, wherein the ECU determines the first and second edge pixels of the first and second detected edges of the first portion of the target and determines the third and fourth edge pixels of the third and fourth detected edges of the second portion of the target by determining a line distance and a line angle based on a pixel gradient and coefficients. ### Claim 17 (depends on 14) The system of claim 14, wherein the ECU determines the first and second vanishing points by performing a J-linkage operation to ensure pair-parallel lines cluster with minimum distance. ### Claim 18 (depends on 14) The system of claim 14, wherein the ECU determines the first and second vanishing points by performing a simulated annealing algorithm. ### Claim 19 (depends on 14) The system of claim 14, wherein the ECU determines, via processing by the image processor of the captured image data, at least two orthogonal vanishing point candidates based on the determined first vanishing point of the first portion of the target and based on the determined second vanishing point of the second portion of the target. ### Claim 20 (depends on 14) The system of claim 14, wherein the ECU determines two orthogonal vanishing point candidates via estimating location of the horizon and zenith. ### Claim 21 A method for calibrating a vehicular vision system, the method comprising: equipping a vehicle with the vehicular vision system, the vehicular vision system comprising at least one vehicular camera disposed at the vehicle; placing a target within a field of view of the vehicular camera, wherein a first portion of the target comprises a first geometric pattern and a second portion of the target comprises a second geometric pattern, and wherein, with the target placed within the field of view of the vehicular camera, the first portion and the second portion are within the field of view of the vehicular camera; capturing image data with the camera, the captured image data representative of the field of view of the vehicular camera; providing the captured image data to an image processor; detecting, via processing at the image processor of the captured image data, first and second edges of the first portion of the target and third and fourth edges of the second portion of the target; determining, via processing at the image processor of the captured image data, first edge pixels representative of the first detected edge of the first portion of the target, second edge pixels representative of the second detected edge of the first portion of the target, third edge pixels representative of the third detected edge of the second portion of the target, and fourth edge pixels representative of the fourth detected edge of the second portion of the target; determining a first vanishing point based on the determined first edge pixels of the first portion of the target and the determined second edge pixels of the first portion of the target, and determining a second vanishing point based on the determined third edge pixels of the second portion of the target and the determined fourth edge pixels of the second portion of the target; determining, via processing by the image processor of the captured image data, at least two orthogonal vanishing point candidates based on the determined first vanishing point of the first portion of the target and based on the determined second vanishing point of the second portion of the target; determining an orientation matrix based on the determined orthogonal vanishing point candidates; determining camera orientation based on location of the determined orientation matrix; and calibrating the vehicular vision system for the vehicular camera based on the determined camera orientation. ### Claim 22 (depends on 21) The method of claim 21, wherein detecting at least one edge of the target comprises reducing a zig-zag effect of the captured image data. ### Claim 23 (depends on 21) The method of claim 21, wherein determining the first and second edge pixels of the first and second detected edges of the first portion of the target and determining the third and fourth edge pixels of the third and fourth detected edges of the second portion of the target comprises grouping the edge pixels of the respective detected edges into respective lines. ### Claim 24 (depends on 21) The method of claim 21, wherein determining the first and second vanishing points comprises performing a J-linkage operation to ensure pair-parallel lines cluster with minimum distance.