# US 10,531,004 B2 · Lidar to camera calibration for generating high definition maps > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,531,004 B2 - Document ID: US10531004B2 - Status: Granted - Kind: B2 - Application number: 16/165,911 - Filed: 2018-10-19 - Published: 2019-04-25 - Granted: 2020-01-07 - Earliest claimed: 2017-10-19 - Assignees: Deepmap Inc. - Applicants: DeepMap Inc. - Inventors: Mark Damon Wheeler; Lin Yang - Web page: https://patentgrep.com/patent/US10531004B2 ## CPC - G01S 7/497: Means for monitoring or calibrating - G01C 21/1652: {with ranging devices, e.g. LIDAR or RADAR} - G01C 21/3602: {Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera} - G01C 25/00: Manufacturing, calibrating, cleaning, or repairing instruments or devices referred to in the other groups of this subclass (testing, calibrating or compensating compasses G01C17/38) - G01S 7/4817: {relating to scanning} - G01S 7/4972: {Alignment of sensor} - G01S 17/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S17/46) - G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders - G01S 17/87: Combinations of systems using electromagnetic waves other than radio waves - G01S 17/89: for mapping or imaging - G01S 17/931: of land vehicles - G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00) - G05D 1/0231: {using optical position detecting means (position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16)} - G05D 1/0248: {in combination with a laser (lasers per se H01S)} - G06T 7/13: Edge detection - G06T 7/33: using feature-based methods - G06T 7/55: from multiple images - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - H04N 5/04: Synchronising (for television systems using pulse code modulation H04N7/56) - H04N 13/106: Processing image signals (for multi-view video sequence encoding H04N19/597) - H04N 23/60: Control of cameras or camera modules - H04N 23/689: {Motion occurring during a rolling shutter mode} - G05D 1/0287: {involving a plurality of land vehicles, e.g. fleet or convoy travelling (traffic control systems for road vehicles G08G1/00, particularly anticollision systems G08G1/16)} - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/10048: Infrared image - G06T 2207/20092: Interactive image processing based on input by user - G06T 2207/20221: Image fusion; Image merging - G06T 2207/30241: Trajectory - G06T 2207/30242: Counting objects in image - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - H04N 23/54: Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A system performs calibration of sensors mounted on a vehicle, for example, lidar and camera sensors mounted on a vehicle, for example, an autonomous vehicle. The system receives a lidar scan and camera image of a view and determines a lidar-to-camera transform based on the lidar scan and the camera image. The system may use a pattern, for example, a checkerboard pattern in the view for calibration. The pattern is placed close to the vehicle to determine an approximate lidar-to-camera transform and then placed at a distance from the vehicle to determine an accurate lidar-to-camera transform. Alternatively, the system determines edges in the lidar scan and the camera image and aligns features based on real-world objects in the scene by comparing edges. ## Claims ### Claim 1 A non-transitory computer readable storage medium storing instructions for performing calibration of sensors of a vehicle, wherein the instructions when executed by a processor, cause the processor to perform the steps including: receiving a first lidar scan of a first view comprising a pattern, the first lidar scan captured by a lidar mounted on an autonomous vehicle, wherein the pattern is positioned less that a first threshold distance from the autonomous vehicle; receiving a first camera image of the first view, the first camera image captured by a camera mounted on the autonomous vehicle; determining an approximate lidar-to-camera transform based on the first lidar scan of the first view and the first camera image of the first view; receiving a second lidar scan of a second view comprising the pattern, the second lidar scan captured by the lidar mounted on the autonomous vehicle, wherein the pattern is positioned greater than a second threshold distance from the autonomous vehicle; receiving, by a camera mounted on the autonomous vehicle, a second camera image of the second view; determining an accurate lidar-to-camera transform based on the location of the pattern in the second lidar scan and the location of the pattern in the camera image of the second view; receiving sensor data comprising images received from the camera and lidar scans from the lidar; generating a high definition map based on the sensor data using the accurate lidar-to-camera transform; and storing the high definition map in a computer readable storage medium for use in navigating the autonomous vehicle. ### Claim 2 (depends on 1) The non-transitory computer readable storage medium of claim 1, wherein the instructions when executed by the processor, further cause the processor to perform steps including: sending signals to the controls of the autonomous vehicle based on the high definition map. ### Claim 3 (depends on 1) The non-transitory computer readable storage medium of claim 1, wherein instructions for determining the accurate lidar-to-camera transform comprises instructions for: detecting location of the pattern in the camera image of the second view; and determining points on the pattern in the second lidar scan based on the approximate lidar-to-camera transform and the points on the pattern in the image of the second view. ### Claim 4 (depends on 1) The non-transitory computer readable storage medium of claim 1, wherein the camera is a left camera, and the camera image is a left camera image, wherein the instructions when executed by the processor, further cause the processor to perform steps including: receiving, by a right camera mounted on the autonomous vehicle, a right camera image of the second view; for a plurality of points on the pattern: detecting a first location of the point from the left camera image; detecting a second location of point from the right camera image; triangulating the first location and the second location to obtain a 3D location of the point in camera coordinates; and determining 3D coordinates of the point by applying the approximate lidar-to-camera transform to the 3D location of the point; fitting a dominant plane within the plurality of points; and adjusting locations of one or more 3D points by projecting the one or more 3D points to the dominant plane. ### Claim 5 (depends on 1) The non-transitory computer readable storage medium of claim 1, wherein the instructions when executed by the processor, further cause the processor to perform steps including: determining a bounding polygon of the pattern using the camera image; projecting a set of 3D points of lidar scan onto the camera image; identifying a subset of 3D points of the lidar scan such that a projected point corresponding to each of the subset of 3D points is within a threshold of the bounding polygon; fitting a dominant plane within the subset of 3D points; and adjusting locations of one or more 3D points by projecting the one or more 3D points to the dominant plane. ### Claim 6 (depends on 1) The non-transitory computer readable storage medium of claim 1, wherein the instructions when executed by the processor, further cause the processor to perform steps including: monitoring portions of viewing area of at least one of the camera or lidar that are covered by the pattern in sensor data captured; and determining a portion of the viewing area that is not covered by the pattern in sensor data captured; and determining the position of the pattern for subsequently capturing sensor data based on the portion of the viewing area that is not covered by the pattern. ### Claim 7 (depends on 6) The non-transitory computer readable storage medium of claim 6, wherein the instructions when executed by the processor, further cause the processor to perform steps including: transmitting information describing the location for placing the pattern corresponding to the determined position. ### Claim 8 (depends on 1) The non-transitory computer readable storage medium of claim 1, wherein the instructions when executed by the processor, further cause the processor to perform steps including: determining position of points on the lidar scan based on intensity data in the lidar scan. ### Claim 9 A method for performing calibration of sensors of a vehicle, the method comprising: receiving a first lidar scan of a first view comprising a pattern, the first lidar scan captured by a lidar mounted on an autonomous vehicle, wherein the pattern is positioned less that a first threshold distance from the autonomous vehicle; receiving a first camera image of the first view, the first camera image captured by a camera mounted on the autonomous vehicle; determining an approximate lidar-to-camera transform based on the first lidar scan of the first view and the first camera image of the first view; receiving a second lidar scan of a second view comprising the pattern, the second lidar scan captured by the lidar mounted on the autonomous vehicle, wherein the pattern is positioned greater than a second threshold distance from the autonomous vehicle; receiving, by a camera mounted on the autonomous vehicle, a second camera image of the second view; determining an accurate lidar-to-camera transform based on the location of the pattern in the second lidar scan and the location of the pattern in the camera image of the second view; receiving sensor data comprising images received from the camera and lidar scans from the lidar; generating a high definition map based on the sensor data using the accurate lidar-to-camera transform; and storing the high definition map in a computer readable storage medium for use in navigating the autonomous vehicle. ### Claim 10 A computer system comprising: one or more processors; and a non-transitory computer readable storage medium storing instructions for performing calibration of sensors of a vehicle, wherein the instructions when executed by a processor, cause the processor to perform the steps including: receiving a first lidar scan of a first view comprising a pattern, the first lidar scan captured by a lidar mounted on an autonomous vehicle, wherein the pattern is positioned less that a first threshold distance from the autonomous vehicle; receiving a first camera image of the first view, the first camera image captured by a camera mounted on the autonomous vehicle; determining an approximate lidar-to-camera transform based on the first lidar scan of the first view and the first camera image of the first view; receiving a second lidar scan of a second view comprising the pattern, the second lidar scan captured by the lidar mounted on the autonomous vehicle, wherein the pattern is positioned greater than a second threshold distance from the autonomous vehicle; receiving, by a camera mounted on the autonomous vehicle, a second camera image of the second view; determining an accurate lidar-to-camera transform based on the location of the pattern in the second lidar scan and the location of the pattern in the camera image of the second view; receiving sensor data comprising images received from the camera and lidar scans from the lidar; generating a high definition map based on the sensor data using the accurate lidar-to-camera transform; and storing the high definition map in a computer readable storage medium for use in navigating the autonomous vehicle.