# US 10,436,885 B2 · Calibrating sensors mounted on an autonomous vehicle > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,436,885 B2 - Document ID: US10436885B2 - Status: Granted - Kind: B2 - Application number: 16/161,035 - Filed: 2018-10-15 - Published: 2019-04-25 - Granted: 2019-10-08 - Earliest claimed: 2017-10-19 - Assignees: Deepmap Inc. - Applicants: DeepMap Inc. - Inventors: Mark Damon Wheeler; Lin Yang - Web page: https://patentgrep.com/patent/US10436885B2 ## CPC - G01S 7/497: Means for monitoring or calibrating - G01C 21/1652: {with ranging devices, e.g. LIDAR or RADAR} - G01C 21/3602: {Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera} - G01C 25/00: Manufacturing, calibrating, cleaning, or repairing instruments or devices referred to in the other groups of this subclass (testing, calibrating or compensating compasses G01C17/38) - G01S 7/4817: {relating to scanning} - G01S 7/4972: {Alignment of sensor} - G01S 17/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S17/46) - G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders - G01S 17/87: Combinations of systems using electromagnetic waves other than radio waves - G01S 17/89: for mapping or imaging - G01S 17/931: of land vehicles - G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00) - G05D 1/0231: {using optical position detecting means (position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16)} - G05D 1/0248: {in combination with a laser (lasers per se H01S)} - G06T 7/13: Edge detection - G06T 7/33: using feature-based methods - G06T 7/55: from multiple images - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - H04N 5/04: Synchronising (for television systems using pulse code modulation H04N7/56) - H04N 13/106: Processing image signals (for multi-view video sequence encoding H04N19/597) - H04N 23/60: Control of cameras or camera modules - H04N 23/689: {Motion occurring during a rolling shutter mode} - G05D 1/0287: {involving a plurality of land vehicles, e.g. fleet or convoy travelling (traffic control systems for road vehicles G08G1/00, particularly anticollision systems G08G1/16)} - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/10048: Infrared image - G06T 2207/20092: Interactive image processing based on input by user - G06T 2207/20221: Image fusion; Image merging - G06T 2207/30241: Trajectory - G06T 2207/30242: Counting objects in image - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - H04N 23/54: Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A system calibrates one or more sensors mounted to an autonomous vehicle. From the one or more sensors, the system identifies a primary sensor and a secondary sensor. The system determines a reference angle for the primary sensor, and based on that reference angle for the primary sensor, a scan-start time representing a start of a scan and a scan-end time representing an end of a scan. The system receives, from the primary sensor, a primary set of scan data recorded from the scan-start time to the scan-end time. The system receives, from the secondary sensor, a secondary set of sensor data recorded from the scan-start time to the scan-end time. The system calibrates the primary and secondary sensors by determining a relative transform for transforming points between the first set of scan data and the second set of scan data. ## Claims ### Claim 1 A non-transitory computer readable storage medium having instructions for calibrating light detection and ranging (LIDAR) sensors mounted on an autonomous vehicle encoded thereon that, when executed by a processor, cause the processor to: identify from a plurality of LIDAR sensors mounted on the autonomous vehicle, a primary LIDAR sensor and a secondary LIDAR sensor; determine a reference angle for the primary LIDAR sensor; determine, based on the reference angle of the primary LIDAR sensor, a scan-start time representing a start of a scan and a scan-end time representing an end of a scan; receive, from the primary LIDAR sensor, a primary set of scan data recorded from the scan-start time to the scan-end time; receive, from the secondary LIDAR sensor, a secondary set of scan data, recorded from the scan-start time to the scan-end time; calibrate the primary and secondary LIDAR sensors by determining a relative transform for transforming points between the primary set of scan data and the secondary set of scan data; generate a high definition map based on scan data recorded by the plurality of LIDAR sensors, the scan data aggregated using the relative transform; and send signals to the controls of the autonomous vehicle based on the high-definition map. ### Claim 2 (depends on 1) The non-transitory computer readable storage medium of claim 1, wherein identifying the primary LIDAR sensor comprises: selecting a LIDAR sensor from the plurality of LIDAR sensors that has a rotational frequency closest to a rate of performing localization for the autonomous vehicle. ### Claim 3 (depends on 1) The non-transitory computer readable storage medium of claim 1, wherein determining the scan-start time and scan-end time comprises: detecting that the primary LIDAR sensor rotates past the reference angle a first time; and determining the scan-start time as the time at which primary LIDAR sensor rotates past the reference angle; detecting that the primary LIDAR sensor rotates past the reference angle a second time, the second time immediately subsequent to the first time; and determining the scan-end time as the time at which primary LIDAR sensor rotates past the reference angle a second time. ### Claim 4 (depends on 1) The non-transitory computer readable storage medium of claim 1, wherein the scan-end time is a scan-start time for a second primary set of scan data recorded by the primary LIDAR sensor. ### Claim 5 (depends on 1) The non-transitory computer readable storage medium of claim 1, further comprising: accessing, from a computer memory, a history of scan data recorded by the secondary LIDAR sensor as the primary LIDAR sensor periodically rotates past the reference angle; and at each scan-start time and scan-end time received from the primary LIDAR sensor, segmenting the history of scan data. ### Claim 6 (depends on 5) The non-transitory computer readable storage medium of claim 5, wherein segmenting the history of scan data comprises: identifying a timestamp assigned to portions of scan data recorded by the secondary LIDAR sensor within a threshold of the scan-start time received from the primary LIDAR sensor. ### Claim 7 (depends on 1) The non-transitory computer readable storage medium of claim 1, wherein the relative transform between the scans of the primary LIDAR sensor and each secondary LIDAR sensor is determined using an iterative closest point technique. ### Claim 8 A computer-implemented method for calibrating sensors mounted on a vehicle, the method comprising: identifying from a plurality of sensors mounted on the vehicle, a primary sensor and a secondary sensor; determining a reference angle for the primary sensor; determining, based on the reference angle of the primary sensor, a scan-start time representing a start of a scan and a scan-end time representing an end of a scan; receiving, from the primary sensor, a primary set of scan data recorded from the scan-start time to the scan-end time; receiving, from the secondary sensor, a secondary set of scan data, recorded from the scan-start time to the scan-end time; calibrating the primary and secondary sensors by determining a relative transform for transforming points between the primary set of scan data and the secondary set of scan data; generating a high definition map based on scan data recorded by the plurality of sensors, the scan data aggregated using the relative transform; and storing the high definition map in a computer readable storage medium. ### Claim 9 (depends on 8) The computer-implemented method of claim 8, wherein identifying the primary sensor comprises: selecting a sensor from the plurality of sensors that has a rotational frequency closest to a rate of performing localization for the vehicle. ### Claim 10 (depends on 8) The computer-implemented method of claim 8, wherein determining the scan-start time and scan-end time comprises: detecting that the primary sensor rotates past the reference angle a first time; and determining the scan-start time as the time at which primary sensor rotates past the reference angle; detecting that the primary sensor rotates past the reference angle a second time, the second time immediately subsequent to the first time; and determining the scan-end time as the time at which sensor rotates past the reference angle a second time. ### Claim 11 (depends on 8) The computer-implemented method of claim 8, wherein the scan-end time is a scan-start time for a second primary set of scan data recorded by the primary sensor. ### Claim 12 (depends on 8) The computer-implemented method of claim 8, further comprising: accessing, from a computer memory, a history of scan data recorded by the secondary sensor as the primary sensor periodically rotates past the reference angle; and at each scan-start time and scan-end time received from the primary sensor, segmenting the history of scan data. ### Claim 13 (depends on 5) The computer-implemented method of claim 5, wherein segmenting the history of scan data comprises: identifying a timestamp assigned to portions of scan data recorded by the secondary sensor within a threshold of the scan-start time received from the primary sensor. ### Claim 14 (depends on 8) The computer-implemented method of claim 8, wherein the relative transform between the scans of the primary sensor and the secondary sensor is determined using an iterative closest point technique. ### Claim 15 A computer-implemented method for calibrating light detection and ranging (LIDAR) sensors mounted on an autonomous vehicle, the method comprising: receiving a reference angle for a primary LIDAR sensor; determining, based on the reference angle of the primary LIDAR sensor, a scan-start time representing a start of a scan and a scan-end time representing an end of a scan; receiving, from the primary LIDAR sensor, a primary set of scan data recorded from the scan-start time to the scan-end time; receiving, from a secondary LIDAR sensor, a secondary set of scan data, recorded from the scan-start time to the scan-end time; calibrating the primary and secondary LIDAR sensors by determining a relative transform for transforming points between the set of scan data and the secondary set of scan data; generating a high definition map based on scan data recorded by the plurality of LIDAR sensors, the scan data aggregated using the relative transform; and sending signals to the controls of the autonomous vehicle based on the high-definition map. ### Claim 16 (depends on 15) The computer-implemented method of claim 15, wherein, determining the scan-start time and scan-end time comprises: detecting that the primary LIDAR sensor rotates past the reference angle a first time; and determining the scan-start time as the time at which primary LIDAR sensor rotates past the reference angle; detecting that the primary LIDAR sensor rotates past the reference angle a second time, the second time immediately subsequent to the first time; and determining the scan-end time as the time at which primary LIDAR sensor rotates past the reference angle a second time. ### Claim 17 (depends on 15) The computer-implemented method of claim 15, wherein the scan-end time is a scan-start time for a second primary set of scan data recorded by the primary LIDAR sensor. ### Claim 18 (depends on 15) The computer-implemented method of claim 15, further comprising: accessing, from a computer memory, a history of scan data recorded by the secondary LIDAR sensor as the primary LIDAR sensor periodically rotates past the reference angle; and at each scan-start time and scan-end time received from the primary LIDAR sensor, segmenting the history of scan data. ### Claim 19 (depends on 18) The computer-implemented method of claim 18, wherein segmenting the history of scan data comprises: identifying a timestamp assigned to portions of scan data recorded by the secondary LIDAR sensor within a threshold of the scan-start time received from the primary LIDAR sensor. ### Claim 20 (depends on 15) The computer-implemented method of claim 15, wherein the relative transform between the scans of the primary LIDAR sensor and each secondary LIDAR sensor is determined using an iterative closest point technique. ### Claim 21 A non-transitory computer readable storage medium storing instructions for calibrating sensors mounted on a vehicle encoded thereon that, when executed by a processor, cause the processor to: identify from a plurality of sensors mounted on the vehicle, a primary sensor and a secondary sensor; determine a reference angle for the primary sensor; determine, based on the reference angle of the primary sensor, a scan-start time representing a start of a scan and a scan-end time representing an end of a scan; receive, from the primary sensor, a primary set of scan data recorded from the scan-start time to the scan-end time; receive, from the secondary sensor, a secondary set of scan data, recorded from the scan-start time to the scan-end time; calibrate the primary and secondary sensors by determining a relative transform for transforming points between the primary set of scan data and the secondary set of scan data; generating a high definition map based on scan data recorded by the plurality of sensors, the scan data aggregated using the relative transform; and store the high definition map in a computer readable storage medium. ### Claim 22 (depends on 21) The non-transitory computer readable storage medium of claim 21, wherein instructions for identifying the primary sensor cause the processor to: select a sensor from the plurality of sensors that has a rotational frequency closest to a rate of performing localization for the vehicle. ### Claim 23 (depends on 21) The non-transitory computer readable storage medium of claim 21, wherein instructions for determining the scan-start time and scan-end time cause the processor to: detect that the primary sensor rotates past the reference angle a first time; and determine the scan-start time as the time at which primary sensor rotates past the reference angle; detect that the primary sensor rotates past the reference angle a second time, the second time immediately subsequent to the first time; and determine the scan-end time as the time at which sensor rotates past the reference angle a second time. ### Claim 24 (depends on 21) The non-transitory computer readable storage medium of claim 21, wherein the scan-end time is a scan-start time for a second primary set of scan data recorded by the primary sensor. ### Claim 25 (depends on 21) The non-transitory computer readable storage medium of claim 21, further comprising instructions encoded thereon that, when executed, further cause the processor to: access, from a computer memory, a history of scan data recorded by the secondary sensor as the primary sensor periodically rotates past the reference angle; and at each scan-start time and scan-end time received from the primary sensor, segment the history of scan data. ### Claim 26 (depends on 25) The non-transitory computer readable storage medium of claim 25, wherein instructions for segmenting the history of scan data cause the processor to: identify a timestamp assigned to portions of scan data recorded by the secondary sensor within a threshold of the scan-start time received from the primary sensor. ### Claim 27 (depends on 21) The non-transitory computer readable storage medium of claim 21, wherein the relative transform between the scans of the primary sensor and the secondary sensor is determined using an iterative closest point technique. ### Claim 28 A computer system for calibrating sensors mounted on a vehicle, the computer system comprising: a processor; and a computer-readable storage medium containing computer program code that when executed, causes the processor to: identify from a plurality of sensors mounted on the vehicle, a primary sensor and a secondary sensor; determine a reference angle for the primary sensor; determine, based on the reference angle of the primary sensor, a scan-start time representing a start of a scan and a scan-end time representing an end of a scan; receive, from the primary sensor, a primary set of scan data recorded from the scan-start time to the scan-end time; receive, from the secondary sensor, a secondary set of scan data, recorded from the scan-start time to the scan-end time; calibrate the primary and secondary sensors by determining a relative transform for transforming points between the primary set of scan data and the secondary set of scan data; generate a high definition map based on scan data recorded by the plurality of sensors, the scan data aggregated using the relative transform; and store the high definition map in a computer readable storage medium. ### Claim 29 (depends on 28) The computer system of claim 28, wherein instructions for identifying the primary sensor cause the processor to: select a sensor from the plurality of sensors that has a rotational frequency closest to a rate of performing localization for the vehicle. ### Claim 30 (depends on 28) The computer system of claim 28, wherein instructions for determining the scan-start time and scan-end time cause the processor to: detect that the primary sensor rotates past the reference angle a first time; and determine the scan-start time as the time at which primary sensor rotates past the reference angle; detect that the primary sensor rotates past the reference angle a second time, the second time immediately subsequent to the first time; and determine the scan-end time as the time at which sensor rotates past the reference angle a second time. ### Claim 31 (depends on 30) The computer system of claim 30, wherein the relative transform between the scans of the primary sensor and the secondary sensor is determined using an iterative closest point technique. ### Claim 32 (depends on 28) The computer system of claim 28, wherein the scan-end time is a scan-start time for a second primary set of scan data recorded by the primary sensor. ### Claim 33 (depends on 28) The computer system of claim 28, further comprising instructions encoded thereon that, when executed, further cause the one or more processors to: access, from a computer memory, a history of scan data recorded by the secondary sensor as the primary sensor periodically rotates past the reference angle; and at each scan-start time and scan-end time received from the primary sensor, segment the history of scan data. ### Claim 34 (depends on 33) The computer system of claim 33, wherein instructions for segmenting the history of scan data cause the processor to: identify a timestamp assigned to portions of scan data recorded by the secondary sensor within a threshold of the scan-start time received from the primary sensor.