# US 2022/0035014 A1 · Dynamic lidar to camera alignment > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0035014 A1 - Document ID: US20220035014A1 - Status: Published application - Kind: A1 - Application number: 16/944,253 - Filed: 2020-07-31 - Published: 2022-02-03 - Earliest claimed: 2020-07-31 - Assignees: GM Global Technology Operations LLC - Applicants: GM Global Technology Operations LLC - Inventors: Yousef A. Omar; Hao Yu; Wende Zhang - Web page: https://patentgrep.com/patent/US20220035014A1 ## CPC - G01S 7/4972: {Alignment of sensor} - G06T 7/73: using feature-based methods - G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders - G01S 17/931: of land vehicles - G06T 7/33: using feature-based methods - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30168: Image quality inspection - G06T 2207/30244: Camera pose - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Systems and method are provided for controlling a vehicle. In one embodiment, a method includes: receiving, by a controller onboard the vehicle, lidar data from the lidar device; receiving, by the controller, image data from the camera device; computing, by the controller, an edge map based on the lidar data; computing, by the controller, an inverse distance transformation (IDT) edge map based on the image data; aligning, by the controller, points of the IDT edge map with points of the lidar edge map to determine extrinsic parameters; storing, by the controller, extrinsic parameters as calibrations in a data storage device; and controlling, by the controller, the vehicle based on the stored calibrations. ## Claims ### Claim 1 A method of controlling a vehicle having a lidar device and a camera device, the method comprising: receiving, by a controller onboard the vehicle, lidar data from the lidar device; receiving, by the controller, image data from the camera device; computing, by the controller, an edge map based on the lidar data; computing, by the controller, an inverse distance transformation (IDT) edge map based on the image data; aligning, by the controller, points of the IDT edge map with points of the lidar edge map to determine extrinsic parameters; storing, by the controller, extrinsic parameters as calibrations in a data storage device; and controlling, by the controller, the vehicle based on the stored calibrations. ### Claim 2 (depends on 1) The method of claim 1, wherein the IDT edge map includes an emphasis on vehicle contours defined in the image data. ### Claim 3 (depends on 1) The method of claim 1, further comprising determining a quality of a pair of points in the IDT edge map and the lidar edge map, and wherein the aligning is based on the quality of the pair. ### Claim 4 (depends on 3) The method of claim 3, further comprising determining the quality of the pair based on a data quality estimation classifier. ### Claim 5 (depends on 3) The method of claim 3, further comprising determining the quality of the pair based on a computed ego-motion of a vehicle. ### Claim 6 (depends on 3) The method of claim 3, further comprising determining the quality of the pair based on a segmentation convolutional neural network. ### Claim 7 (depends on 1) The method of claim 1, wherein the aligning is based on a search method that computes six degrees of freedom of objects in the IDT edge map and the lidar edge map. ### Claim 8 (depends on 7) The method of claim 7, wherein the search method comprises a two-stage search method, the first stage comprising a coarse level of refinement, and the second stage comprising a fine level of refinement. ### Claim 9 (depends on 7) The method of claim 7, wherein the aligning is iteratively performed until a confidence estimation reaches a predefined percentage. ### Claim 10 (depends on 7) The method of claim 7, wherein the aligning is iteratively performed until a count of iterations reaches a predefined value. ### Claim 11 A vehicle system of a vehicle, comprising: a lidar device; a camera device; and a controller configured to, by a processor, receive lidar data from the lidar device, receive image data from the camera device, compute an edge map based on the lidar data, compute an inverse distance transformation (IDT) edge map based on the image data, align points of the IDT edge map with points of the lidar edge map to determine extrinsic parameters, store extrinsic parameters as calibrations in a data storage device, and control the vehicle based on the stored calibrations. ### Claim 12 (depends on 11) The system of claim 11, wherein the IDT edge map includes an emphasis on vehicle contours defined in the image data. ### Claim 13 (depends on 11) The system of claim 11, wherein the controller is further configured to determine a quality of a pair of points in the IDT edge map and the lidar edge map, and wherein the aligning is based on the quality of the pair. ### Claim 14 (depends on 13) The system of claim 13, wherein the controller is further configured to determine the quality of the pair based on a data quality estimation classifier. ### Claim 15 (depends on 13) The system of claim 13, wherein the controller is further configured to determine the quality of the pair based on a computed ego-motion of a vehicle. ### Claim 16 (depends on 13) The system of claim 13, wherein the controller is further configured to determine the quality of the pair based on a segmentation convolutional neural network. ### Claim 17 (depends on 11) The system of claim 11, wherein the controller is configured to align based on a search method that computes six degrees of freedom of objects in the IDT edge map. ### Claim 18 (depends on 17) The system of claim 17, wherein the search method comprises a two-stage search method, the first stage comprising a coarse level of refinement, and the second stage comprising a fine level of refinement. ### Claim 19 (depends on 17) The system of claim 17, wherein controller is configured to iteratively align until a confidence estimation reaches a predefined percentage. ### Claim 20 (depends on 17) The system of claim 17, wherein the controller is configured to iteratively align until a count of iterations reaches a predefined value.