# US 12,118,732 B2 · Systems and methods for object detection with LiDAR decorrelation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,732 B2 - Document ID: US12118732B2 - Status: Granted - Kind: B2 - Application number: 17/101,313 - Filed: 2020-11-23 - Published: 2024-06-06 - Granted: 2024-10-15 - Earliest claimed: 2020-11-23 - Assignees: Ford Global Technologies, LLC - Applicants: Ford Global Technologies, LLC - Inventors: Kevin Lee Wyffels - Web page: https://patentgrep.com/patent/US12118732B2 ## CPC - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders - G01S 17/89: for mapping or imaging - G01S 17/931: of land vehicles - G06F 18/2185: {the supervisor being an automated module, e.g. intelligent oracle} - G06F 18/24133: {Distances to prototypes} - G06T 7/11: Region-based segmentation - G06V 10/82: using neural networks - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Systems/methods for operating an autonomous vehicle. The methods comprise: obtaining LiDAR data generated by a LiDAR system of the autonomous vehicle and image data generated by a camera of the autonomous vehicle; performing a first object detection algorithm to generate first object detection information using the LiDAR data, and a second object detection algorithm to generate second object detection information using both the LiDAR data and the image data; and processing the first and second object detection information to determine whether a given object was detected by both the first and second object detection algorithms. When a determination is made that the given object was detected by both the first and second object detection algorithms, the first object detection information is selectively modified based on contents of the second object detection information. The modified first object detection information is used to facilitate at least one autonomous driving operation. ## Claims ### Claim 1 A method for operating an autonomous vehicle, comprising: obtaining, by a computing device, LiDAR data generated by a LiDAR system of the autonomous vehicle and image data generated by a camera of the autonomous vehicle; performing, by the computing device, a first object detection algorithm to generate first object detection information using the LiDAR data, and a second object detection algorithm to generate second object detection information using both the LiDAR data and the image data; processing, by the computing device, the first and second object detection information to determine whether a given object was detected by both the first and second object detection algorithms; performing, by the computing device, the following operations when a determination is made that the given object was detected by both the first and second object detection algorithms; selectively modifying the first object detection information based on contents of the second object detection information; and using, by the computing device, the modified first object detection information to facilitate at least one autonomous driving operation. ### Claim 2 (depends on 1) The method according to claim 1, further comprising using, by the computing device, the second object detection information to facilitate at least one autonomous driving operation, when a determination is made that the given object was not detected by both the first and second object detection algorithms. ### Claim 3 (depends on 1) The method according to claim 1, wherein the first object detection algorithm comprises a closed world object detection algorithm implemented by a neural network that is trained to detect particular types of objects. ### Claim 4 (depends on 3) The method according to claim 3, wherein the second object detection algorithm comprises an open world object detection algorithm that is configured to detect any type of object. ### Claim 5 (depends on 4) The method according to claim 4, wherein the open world detection algorithm comprises camera-LiDAR fusion object detection. ### Claim 6 (depends on 4) The method according to claim 4, wherein the first object detection information is selectively modified based on (a) a total number of data points Nopen of the LiDAR data that are contained in a point cloud segment Sopen defined for an object detected by the open world detection algorithm and (b) a total number of data points Nclosed of the LiDAR data that are contained in a point cloud segment Sclosed defined for an object detected by the closed world detection algorithm. ### Claim 7 (depends on 6) The method according to claim 6, wherein a size of the point cloud segment Sclosed is increased when the total number of data points Nopen is greater than the total number of data points Nclosed. ### Claim 8 (depends on 6) The method according to claim 6, wherein a size of the point cloud segment Sclosed is decreased when the total number of data points Nopen is less than the total number of data points Nclosed. ### Claim 9 (depends on 1) The method according to claim 1, further comprising discarding the first object detection information such that the first object detection information is not used to facilitate the at least one autonomous driving operation. ### Claim 10 (depends on 1) The method according to claim 1, further comprising: determining whether the first object detection information should be modified based on a difference in content between the first and second object detection information; wherein the first object detection information is selectively modified only when a determination is made that the first object detection information should be modified. ### Claim 11 A system, comprising: a processor; a non-transitory computer-readable storage medium comprising programming instructions that are configured to cause the processor to implement a method for operating an autonomous vehicle, wherein the programming instructions comprise instructions to: obtain LiDAR data generated by a LiDAR system of the autonomous vehicle and image data generated by a camera of the autonomous vehicle; perform a first object detection algorithm to generate first object detection information using the LiDAR data, and a second object detection algorithm to generate second object detection information using both the LiDAR data and the image data; process the first and second object detection information to determine whether a given object was detected by both the first and second object detection algorithms; perform the following operations when a determination is made that the given object was detected by both the first and second object detection algorithms; selectively modify the first object detection information based on contents of the second object detection information; and causing the modified first object detection information to be used for at least one autonomous driving operation. ### Claim 12 (depends on 11) The system according to claim 11, wherein the programming instructions further comprise instructions to cause the second object detection information to be used at least one autonomous driving operation, when a determination is made that the given object was not detected by both the first and second object detection algorithms. ### Claim 13 (depends on 11) The system according to claim 11, wherein the first object detection algorithm comprises a closed world object detection algorithm implemented by a neural network that is trained to detect particular types of objects. ### Claim 14 (depends on 13) The system according to claim 13, wherein the second object detection algorithm comprises an open world object detection algorithm that is configured to detect any type of object. ### Claim 15 (depends on 14) The system according to claim 14, wherein the open world detection algorithm comprises camera-LiDAR fusion object detection. ### Claim 16 (depends on 14) The system according to claim 14, wherein the first object detection information is selectively modified based on (a) a total number of data points Nopen of the LiDAR data that are contained in a point cloud segment Sopen defined for an object detected by the open world detection algorithm and (b) a total number of data points Nclosed of the LiDAR data that are contained in a point cloud segment Sclosed defined for an object detected by the closed world detection algorithm. ### Claim 17 (depends on 16) The system according to claim 16, wherein a size of the point cloud segment Sclosed is increased when the total number of data points Nopen is greater than the total number of data points Nclosed. ### Claim 18 (depends on 16) The system according to claim 16, wherein a size of the point cloud segment Sclosed is decreased when the total number of data points Nopen is less than the total number of data points Nclosed. ### Claim 19 (depends on 11) The system according to claim 11, wherein the programming instructions further comprise instructions to discard the first object detection information such that the first object detection information is not used to facilitate the at least one autonomous driving operation. ### Claim 20 (depends on 11) The system according to claim 11, wherein the programming instructions further comprise instructions to: determine whether the first object detection information should be modified based on a difference in content between the first and second object detection information; wherein the first object detection information is selectively modified only when a determination is made that the first object detection information should be modified.