# US 12,008,777 B2 · Validating an SfM map using lidar point clouds > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,777 B2 - Document ID: US12008777B2 - Status: Granted - Kind: B2 - Application number: 17/508,681 - Filed: 2021-10-22 - Published: 2023-04-27 - Granted: 2024-06-11 - Earliest claimed: 2021-10-22 - Assignees: Argo AI, LLC - Applicants: Argo Ai, LLC - Inventors: Kunal Anil Desai; Xxx Xinjilefu; Gang Pan; Manu Sethi; Tao V. Fu - Web page: https://patentgrep.com/patent/US12008777B2 ## CPC - G06T 7/579: from motion - G01S 7/4808: {Evaluating distance, position or velocity data} - G01S 7/4815: {using multiple transmitters} - G01S 7/4816: {of receivers alone} - G01S 17/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S17/46) - G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders - G01S 17/89: for mapping or imaging - G01S 17/894: Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar - G06T 5/70: Denoising; Smoothing - G01S 7/499: using polarisation effects - G01S 17/931: of land vehicles - G06T 2200/04: involving 3D image data - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES ## Abstract Disclosed herein are system and method embodiments to implement a validation of an SfM map. An embodiment operates by receiving a motion-generated map corresponding to a digital image, generating a first depth map, wherein the first depth map comprises depth information for one or more triangulated points located within the motion generated image. The embodiment further receives a light detection and ranging (lidar) generated point cloud including at least a portion of the one or more triangulated points, splats the lidar point cloud proximate to the portion of the one or more triangulated points and generates a second depth map for the portion and identifies an incorrect triangulated point, of the one or more triangulated points, based on comparing the first depth information to the second depth information. The incorrect triangulated points may be removed from the SfM map or marked with a low degree of confidence. ## Claims ### Claim 1 A method of validating triangulated points in a digital image, the method comprising, receiving a motion-generated image corresponding to the digital image; generating a first depth map, wherein the first depth map comprises first depth information for one or more triangulated points located within the motion-generated image; receiving a light detection and ranging (lidar) generated point cloud comprising a portion of the one or more triangulated points; splatting the lidar generated point cloud proximate to the portion of the one or more triangulated points to generate a splatted point cloud; generating a second depth map for at least the portion of the one or more triangulated points, wherein the second depth map comprises second depth information for the portion of the one or more triangulated points located within the splatted point cloud; and validating a triangulated point, of the portion of one or more triangulated points, based on comparing the first depth information to the second depth information. ### Claim 2 (depends on 1) The method of claim 1, wherein the motion-generated image comprises a three-dimensional map generated by Structure from Motion (SfM) imaging. ### Claim 3 (depends on 2) The method of claim 2, wherein the SfM imaging is generated from a plurality of images comprising one or more common key points. ### Claim 4 (depends on 1) The method of claim 1, wherein validating the triangulated point further comprises identifying an incorrect triangulated point and removing the incorrect triangulated point from the first depth map. ### Claim 5 (depends on 4) The method of claim 4, further comprising removing the incorrect triangulated point from the motion-generated image. ### Claim 6 (depends on 4) The method of claim 4, wherein the identifying the incorrect triangulated point further comprises determining whether the second depth information does not favorably compare, within a predetermined threshold, to the first depth information. ### Claim 7 (depends on 1) The method of claim 1, wherein validating the triangulated point further comprises determining one or more correct triangulated points, when the second depth information compares favorably, within a predetermined threshold, to the first depth information. ### Claim 8 (depends on 7) The method of claim 7, wherein the determining one or more correct triangulated points further comprises calculating consensus depth information for a plurality of the one or more triangulated points. ### Claim 9 (depends on 7) The method of claim 7, further comprising determining a confidence for the one or more correct triangulated points. ### Claim 10 (depends on 1) The method of claim 1, wherein the generating the first depth map further comprises comparing multiple camera poses for a plurality of camera images. ### Claim 11 (depends on 1) The method of claim 1, wherein the generating the first depth map further comprises comparing depths from the first depth map of triangulated points against depths from triangulating these points. ### Claim 12 (depends on 1) The method of claim 1, wherein the splatting comprises rendering opaque surfaces from points within the lidar generated point cloud without connectivity. ### Claim 13 (depends on 1) The method of claim 1, wherein the splatting comprises determining a neighborhood around key points of the lidar generated point cloud, estimating a surface normal at each of the key points, computing splats with varying radii and rendering the computed splats. ### Claim 14 A system, comprising: a memory; and at least once processor coupled to the memory and configured to: receive a motion-generated image corresponding to a digital image; generate a first depth map, wherein the first depth map comprises first depth information for one or more triangulated points located within the motion-generated image; receive a light detection and ranging (lidar) generated point cloud comprising a portion of the one or more triangulated points; splat the lidar generated point cloud proximate to the portion of the one or more triangulated points to generate a splatted point cloud; generate a second depth map for at least the portion of the one or store triangulated points, wherein the second depth map comprises second depth information for the portion of the one or more triangulated points located within the splatted point cloud; and validate a triangulated point, of the portion of one or more triangulated points, based on comparing the first depth information to the second depth information. ### Claim 15 (depends on 14) The system of claim 14, wherein the at least one processor is further configured to identify the validated triangulated point as an incorrect triangulated point when the second depth information does not favorably compare, within a predetermined threshold, to the first depth information. ### Claim 16 (depends on 14) The system of claim 14, wherein the at least one processor is further configured to identify the validated triangulated point as a correct triangulated point when the second depth information favorably compares, within a predetermined threshold, to the first depth information. ### Claim 17 (depends on 15) The system of claim 15, wherein the at least one processor is further configured to determine consensus depth information for one or more of the correct triangulated points. ### Claim 18 (depends on 14) The system of claim 14, wherein the at least one processor is further configured to determine a confidence for the correct triangulated point. ### Claim 19 (depends on 14) The system of claim 14, wherein the wherein the splat comprises: rendering opaque surfaces from points within the lidar generated point cloud without connectivity. ### Claim 20 A method of correcting a structure for motion (SfM) map comprising: receiving a structure for motion-generated map; generating a first depth map, wherein the first depth map comprises first depth information for one or more triangulated points located within the structure for motion-generated map; receiving a light detection and ranging (lidar) generated point cloud comprising a portion of the one or more triangulated points; splatting the lidar generated point cloud proximate to the portion of the one or more triangulated points to generate a splatted point cloud; generating a second depth map for at least the portion of the one or more triangulated points, wherein the second depth map comprises second depth information for the portion of the one or more triangulated points located within the splatted point cloud; identifying an incorrect triangulated point, of the portion of one or more triangulated points, based on comparing the first depth information to the second depth information; and removing the incorrect triangulated point from the structure for motion-generated map.