# US 10,999,511 B2 · LIDAR and camera synchronization > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,999,511 B2 - Document ID: US10999511B2 - Status: Granted - Kind: B2 - Application number: 16/580,969 - Filed: 2019-09-24 - Published: 2020-01-16 - Granted: 2021-05-04 - Earliest claimed: 2017-10-19 - Assignees: Deepmap Inc. - Applicants: DeepMap Inc. - Inventors: Lin Yang; Mark Damon Wheeler - Web page: https://patentgrep.com/patent/US10999511B2 ## CPC - G01S 7/497: Means for monitoring or calibrating - G01C 21/1652: {with ranging devices, e.g. LIDAR or RADAR} - G01C 21/3602: {Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera} - G01C 25/00: Manufacturing, calibrating, cleaning, or repairing instruments or devices referred to in the other groups of this subclass (testing, calibrating or compensating compasses G01C17/38) - G01S 7/4817: {relating to scanning} - G01S 7/4972: {Alignment of sensor} - G01S 17/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S17/46) - G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders - G01S 17/87: Combinations of systems using electromagnetic waves other than radio waves - G01S 17/89: for mapping or imaging - G01S 17/931: of land vehicles - G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00) - G05D 1/0231: {using optical position detecting means (position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16)} - G05D 1/0248: {in combination with a laser (lasers per se H01S)} - G06T 7/13: Edge detection - G06T 7/33: using feature-based methods - G06T 7/55: from multiple images - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - H04N 5/04: Synchronising (for television systems using pulse code modulation H04N7/56) - H04N 13/106: Processing image signals (for multi-view video sequence encoding H04N19/597) - H04N 23/60: Control of cameras or camera modules - H04N 23/689: {Motion occurring during a rolling shutter mode} - G05D 1/0287: {involving a plurality of land vehicles, e.g. fleet or convoy travelling (traffic control systems for road vehicles G08G1/00, particularly anticollision systems G08G1/16)} - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/10048: Infrared image - G06T 2207/20092: Interactive image processing based on input by user - G06T 2207/20221: Image fusion; Image merging - G06T 2207/30241: Trajectory - G06T 2207/30242: Counting objects in image - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - H04N 23/54: Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method and system for synchronizing a lidar and a camera on an autonomous vehicle. The system instructs the camera to detect light columns transmitted by the lidar. The system iterates through various start times for the camera. The system instructs the lidar to emit a plurality of light columns at a lidar frequency. The system instructs the camera to capture images at a camera frequency starting at each start time. The system analyzes the image data received from the cameras to identify light columns captured in the images. The system calculates an alignment score for each of the many start times based on the identified light columns. The start time with the optimal alignment score is selected and used to synchronize the lidar and the camera. With lidar data detected by the synchronized lidar and image data captured by the synchronized camera, the system may navigate the autonomous vehicle. ## Claims ### Claim 1 A method for synchronizing a light detection and ranging sensor (lidar) and a camera on an autonomous vehicle, comprising: transmitting, via the lidar, a plurality of light columns at a lidar frequency within a field of view (FOV) of the camera; for each start time of a plurality of start times: capturing, via the camera, a plurality of images at a camera frequency, each image captured over an exposure time, for each image of the plurality of images, identifying a number of light columns in the image, and calculating an alignment score based on the identified number of light columns in each of the plurality of images; selecting a start time with the highest alignment score; synchronizing the lidar and the camera according to the selected start time; and navigating the autonomous vehicle with lidar data detected by the synchronized lidar and image data captured by the synchronized camera. ### Claim 2 (depends on 1) The method of claim 1, wherein the plurality of light columns comprises at least three light columns within the FOV of the camera. ### Claim 3 (depends on 1) The method of claim 1, wherein the lidar emits the light columns in an infrared spectrum, and wherein the camera detects light in at least the infrared spectrum. ### Claim 4 (depends on 1) The method of claim 1, wherein for each start time, for each image, identifying the number of light columns in the image, further comprises: transforming each pixel in the image into an intensity value; calculating a gradient strength for each pixel based on intensity values of neighboring pixels; and identifying one or more pixels above a threshold gradient strength; wherein the number of light columns in the image is according to the identified one or more pixels. ### Claim 5 (depends on 1) The method of claim 1, wherein the lidar emits the light columns towards a reflective surface within a threshold distance from the lidar and the camera. ### Claim 6 (depends on 5) The method of claim 5, wherein the reflective surface is sufficiently wide to be incident with a threshold number of light columns. ### Claim 7 (depends on 1) The method of claim 1, wherein the plurality of start times is determined based on the exposure time for the camera and the camera frequency. ### Claim 8 (depends on 7) The method of claim 7, wherein a number of start times for the plurality of start times is based on a ratio of the exposure time for the camera to the camera frequency. ### Claim 9 (depends on 7) The method of claim 7, wherein a first start time and a second start time of the plurality of start times are within the exposure time for the camera. ### Claim 10 (depends on 1) The method of claim 1, wherein a number of images for the plurality of images is determined based on the camera frequency and the lidar frequency. ### Claim 11 (depends on 1) The method of claim 1, wherein for each start time, calculating the alignment score based on the determination in each of the plurality of images, further comprises: counting a sum of light columns determined to be present in the plurality of images; wherein the alignment score is based on the sum of light columns. ### Claim 12 (depends on 1) The method of claim 1, wherein synchronizing the lidar and the camera comprises adjusting one or more of: a lidar clock on the lidar and a camera clock on the camera. ### Claim 13 A non-transitory computer-readable storage medium for synchronizing a light detection and ranging sensor (lidar) and a camera on an autonomous vehicle, the non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising: transmitting, via the lidar, a plurality of light columns at a lidar frequency within a field of view (FOV) of the camera; for each start time of a plurality of start times: capturing, via the camera, a plurality of images at a camera frequency, each image captured over an exposure time, for each image of the plurality of images, identifying a number of light columns in the image, and calculating an alignment score based on the identified number of light columns in each of the plurality of images; selecting a start time with the highest alignment score; synchronizing the lidar and the camera according to the selected start time; and navigating the autonomous vehicle with lidar data detected by the synchronized lidar and image data captured by the synchronized camera. ### Claim 14 (depends on 13) The non-transitory computer-readable storage medium of claim 13, wherein the plurality of light columns comprises at least three light columns within the FOV of the camera. ### Claim 15 (depends on 13) The non-transitory computer-readable storage medium of claim 13, wherein the lidar emits the light columns in an infrared spectrum, and wherein the camera detects light in at least the infrared spectrum. ### Claim 16 (depends on 13) The non-transitory computer-readable storage medium of claim 13, wherein for each start time, for each image, identifying the number of light columns in the image, further comprises: transforming each pixel in the image into an intensity value; calculating a gradient strength for each pixel based on intensity values of neighboring pixels; and identifying one or more pixels above a threshold gradient strength; wherein the number of light columns in the image is according to the identified one or more pixels. ### Claim 17 (depends on 13) The non-transitory computer-readable storage medium of claim 13, wherein the lidar emits the light columns towards a reflective surface within a threshold distance from the lidar and the camera. ### Claim 18 (depends on 17) The non-transitory computer-readable storage medium of claim 17, wherein the reflective surface is sufficiently wide to be incident with a threshold number of light columns. ### Claim 19 (depends on 13) The non-transitory computer-readable storage medium of claim 13, wherein the plurality of start times is determined based on the exposure time for the camera and the camera frequency. ### Claim 20 (depends on 19) The non-transitory computer-readable storage medium of claim 19, wherein a number of start times for the plurality of start times is based on a ratio of the exposure time for the camera to the camera frequency. ### Claim 21 (depends on 19) The non-transitory computer-readable storage medium of claim 19, wherein a first start time and a second start time of the plurality of start times are within the exposure time for the camera. ### Claim 22 (depends on 13) The non-transitory computer-readable storage medium of claim 13, wherein a number of images for the plurality of images is determined based on the camera frequency and the lidar frequency. ### Claim 23 (depends on 13) The non-transitory computer-readable storage medium of claim 13, wherein for each start time, calculating the alignment score based on the determination in each of the plurality of images, further comprises: counting a sum of light columns determined to be present in the plurality of images; wherein the alignment score is based on the sum of light columns. ### Claim 24 (depends on 13) The non-transitory computer-readable storage medium of claim 13, wherein synchronizing the lidar and the camera comprises adjusting one or more of: a lidar clock on the lidar and a camera clock on the camera. ### Claim 25 A system comprising: a lidar on an autonomous vehicle; a camera on the autonomous vehicle; a processor; and a non-transitory computer-readable storage medium for synchronizing a light detection and ranging sensor (lidar) and a camera on an autonomous vehicle, the non-transitory computer-readable storage medium storing instructions that, when executed by the processor, cause the processor to perform operations comprising: transmitting, via the lidar, a plurality of light columns at a lidar frequency within a field of view (FOV) of the camera; for each start time of a plurality of start times: capturing, via the camera, a plurality of images at a camera frequency, each image captured over an exposure time, for each image of the plurality of images, identifying a number of light columns in the image, and calculating an alignment score based on the identified number of light columns in each of the plurality of images; selecting a start time with the highest alignment score; synchronizing the lidar and the camera according to the selected start time; and navigating the autonomous vehicle with lidar data detected by the synchronized lidar and image data captured by the synchronized camera. ### Claim 26 (depends on 25) The system of claim 25, wherein the plurality of light columns comprises at least three light columns within the FOV of the camera. ### Claim 27 (depends on 25) The system of claim 25, wherein the lidar emits the light columns in an infrared spectrum, and wherein the camera detects light in at least the infrared spectrum. ### Claim 28 (depends on 25) The system of claim 25, wherein for each start time, for each image, identifying the number of light columns in the image, further comprises: transforming each pixel in the image into an intensity value; calculating a gradient strength for each pixel based on intensity values of neighboring pixels; and identifying one or more pixels above a threshold gradient strength; wherein the number of light columns in the image is according to the identified one or more pixels. ### Claim 29 (depends on 25) The system of claim 25, wherein for each start time, calculating the alignment score based on the determination in each of the plurality of images, further comprises: counting a sum of light columns determined to be present in the plurality of images; wherein the alignment score is based on the sum of light columns. ### Claim 30 (depends on 25) The system of claim 25, wherein synchronizing the lidar and the camera comprises adjusting one or more of: a lidar clock on the lidar and a camera clock on the camera.