# US 12,112,497 B2 · LIDAR-camera fusion where LIDAR and camera validly see different things > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,497 B2 - Document ID: US12112497B2 - Status: Granted - Kind: B2 - Application number: 17/809,612 - Filed: 2022-06-29 - Published: 2022-10-13 - Granted: 2024-10-08 - Earliest claimed: 2020-01-03 - Assignees: Mobileye Vision Technologies Ltd. - Applicants: Mobileye Vision Technologies Ltd. - Inventors: Chaim Blau; Ofer Springer; Kevin Rosenblum; Alon Ziv; Erez Dagan; David Boublil; Nadav Shaag; David Neuhof; Jeffrey Moskowitz; Gal Topel; Yotam Stern - Web page: https://patentgrep.com/patent/US12112497B2 ## CPC - G01C 21/3848: {Data obtained from both position sensors and additional sensors} - G06T 7/55: from multiple images - B60W 10/04: including control of propulsion units - B60W 10/18: including control of braking systems - B60W 10/20: including control of steering systems - B60W 30/09: Taking automatic action to avoid collision, e.g. braking and steering - B60W 60/001: {Planning or execution of driving tasks} - G01B 11/22: for measuring depth - G01C 21/1652: {with ranging devices, e.g. LIDAR or RADAR} - G01C 21/1656: {with passive imaging devices, e.g. cameras} - G01C 21/30: Map- or contour-matching - G01C 21/3841: {Data obtained from two or more sources, e.g. probe vehicles} - G01C 21/3885: {Transmission of map data to client devices; Reception of map data by client devices} - G01S 7/4817: {relating to scanning} - G01S 17/42: Simultaneous measurement of distance and other co-ordinates (indirect measurement G01S17/46) - G01S 17/58: Velocity or trajectory determination systems; Sense-of-movement determination systems - G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders - G01S 17/89: for mapping or imaging - G01S 17/931: of land vehicles - H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture - H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2420/408: Radar; Laser, e.g. lidar - B60W 2554/20: Static objects - B60W 2554/4042: Longitudinal speed - B60W 2554/802: Longitudinal distance - B60W 2556/45: External transmission of data to or from the vehicle - B60W 2720/10: Longitudinal speed - B60W 2720/24: Direction of travel - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A navigation system for a host vehicle may include a processor programmed to: receive from a camera onboard the host vehicle at least one captured image representative of an environment of the host vehicle, wherein the camera is positioned at a first location relative to the host vehicle; receive point cloud information from a LIDAR system onboard the host vehicle, wherein the LIDAR system is positioned at a second location relative to the host vehicle; analyze the at least one captured image and the received point cloud information to detect one or more objects in the shared field of view region; determine whether a vantage point difference between the first location of the camera and the second location of the LIDAR system accounts for the one or more detected objects being represented in only one of the at least one captured image or the received point cloud information. ## Claims ### Claim 1 A navigation system for a host vehicle, the navigation system comprising: at least one processor programmed to: receive from a camera onboard the host vehicle at least one captured image representative of an environment of the host vehicle, wherein the camera is positioned at a first location relative to the host vehicle; receive point cloud information from a LIDAR system onboard the host vehicle, wherein the point cloud information is representative of distances to various objects in an environment of the host vehicle, wherein the LIDAR system is positioned at a second location relative to the host vehicle, wherein the second location is different from the first location, and wherein a field of view of the camera at least partially overlaps with a field of view of the LIDAR system to provide a shared field of view region; analyze the at least one captured image and the received point cloud information to detect one or more objects in the shared field of view region, wherein the detected one or more objects are represented in only one of the at least one captured image or the received point cloud information; determine whether a vantage point difference between the first location of the camera and the second location of the LIDAR system accounts for the one or more detected objects being represented in only one of the at least one captured image or the received point cloud information; if the vantage point difference does not account for the one or more detected objects being represented in only one of the at least one captured image or the received point cloud information, then cause at least one remedial action to be taken; and if the vantage point difference does account for the one or more detected objects being represented in only one of the at least one captured image or the received point cloud information, then determine at least one navigational action to be taken by the host vehicle based on the one or more detected objects and cause the at least one navigational action to be taken by the host vehicle. ### Claim 2 (depends on 1) The navigation system of claim 1, wherein the at least one navigational action includes at least one of maintaining a current heading direction and speed for the host vehicle, changing a current heading direction for the host vehicle, or changing a speed of the host vehicle. ### Claim 3 (depends on 1) The navigation system of claim 1, wherein the at least one processor is programmed to cause the at least one navigational action to be taken by the host vehicle by causing activation of one or more actuators associated with a steering system, a braking system, or a drive system of the host vehicle. ### Claim 4 (depends on 1) The navigation system of claim 1, wherein the one or more detected objects include an object located ahead of a target vehicle forward of the host vehicle. ### Claim 5 (depends on 4) The navigation system of claim 4, wherein the object located ahead of the target vehicle includes a static object in a roadway. ### Claim 6 (depends on 4) The navigation system of claim 4, wherein the object located ahead of the target vehicle includes a second target vehicle. ### Claim 7 (depends on 6) The navigation system of claim 6, wherein the at least one navigational action is determined based on a detected action of the second target vehicle. ### Claim 8 (depends on 7) The navigation system of claim 7, wherein the at least one navigational action includes braking the host vehicle based on a detected slowing of the second target vehicle. ### Claim 9 (depends on 8) The navigation system of claim 8, wherein the target vehicle is not detected to be slowing. ### Claim 10 (depends on 1) The navigation system of claim 1, wherein the one or more detected objects include an object located ahead of the host vehicle but close enough that a portion of the host vehicle obscures the one or more objects from either the at least one captured image or the point cloud information. ### Claim 11 (depends on 10) The navigation system of claim 10, wherein the portion of the host vehicle includes a hood of the host vehicle. ### Claim 12 (depends on 1) The navigation system of claim 1, wherein the camera is located behind a windshield of the host vehicle, and the LIDAR system is mounted on a roof of the host vehicle. ### Claim 13 (depends on 1) The navigation system of claim 1, wherein the camera is located in a vicinity of a front bumper of the host vehicle, and the LIDAR system is mounted on a roof of the host vehicle. ### Claim 14 (depends on 1) The navigation system of claim 1, wherein the camera is located behind a windshield of the host vehicle, and the LIDAR system is located in a vicinity of a front bumper of the host vehicle. ### Claim 15 (depends on 1) The navigation system of claim 1, wherein the at least one remedial action includes issuing a warning to a passenger of the host vehicle. ### Claim 16 (depends on 1) The navigation system of claim 1, wherein the at least one remedial action includes disregarding the one or more detected objects in navigation of the host vehicle. ### Claim 17 (depends on 1) The navigation system of claim 1, wherein the at least one remedial action includes applying different weights to an image-based detection path based on analysis of the at least one captured image and a LIDAR-based detection path based on analysis of the point cloud information. ### Claim 18 (depends on 17) The navigation system of claim 17, wherein the different weights are determined based on detected weather conditions. ### Claim 19 (depends on 17) The navigation system of claim 17, wherein the different weights are determined based on detected lighting conditions. ### Claim 20 (depends on 1) The navigation system of claim 1, wherein the point cloud information includes multiple, captured LIDAR frames. ### Claim 21 (depends on 1) The navigation system of claim 1, wherein the at least one captured image includes a plurality of captured images. ### Claim 22 A method for determining a navigational action for a host vehicle, the method comprising: receiving from a camera onboard the host vehicle at least one captured image representative of an environment of the host vehicle, wherein the camera is positioned at a first location relative to the host vehicle; receiving point cloud information from a LIDAR system onboard the host vehicle, wherein the point cloud information is representative of distances to various objects in an environment of the host vehicle, wherein the LIDAR system is positioned at a second location relative to the host vehicle, wherein the second location is different from the first location, and wherein a field of view of the camera at least partially overlaps with a field of view of the LIDAR system to provide a shared field of view region; analyzing the at least one captured image and the received point cloud information to detect one or more objects in the shared field of view region, wherein the detected one or more objects are represented in only one of the at least one captured image or the received point cloud information; determining whether a vantage point difference between the first location of the camera and the second location of the LIDAR system accounts for the one or more detected objects being represented in only one of the at least one captured image or the received point cloud information; if the vantage point difference does not account for the one or more detected objects being represented in only one of the at least one captured image or the received point cloud information, then causing at least one remedial action to be taken; and if the vantage point difference does account for the one or more detected objects being represented in only one of the at least one captured image or the received point cloud information, then determining at least one navigational action to be taken by the host vehicle based on the one or more detected objects and cause the at least one navigational action to be taken by the host vehicle. ### Claim 23 A computer-readable medium storing instructions that when executed by at least one processor, are configured to cause the at least one processor programmed to: receive from a camera onboard the host vehicle at least one captured image representative of an environment of the host vehicle, wherein the camera is positioned at a first location relative to the host vehicle; receive point cloud information from a LIDAR system onboard the host vehicle, wherein the point cloud information is representative of distances to various objects in an environment of the host vehicle, wherein the LIDAR system is positioned at a second location relative to the host vehicle, wherein the second location is different from the first location, and wherein a field of view of the camera at least partially overlaps with a field of view of the LIDAR system to provide a shared field of view region; analyze the at least one captured image and the received point cloud information to detect one or more objects in the shared field of view region, wherein the detected one or more objects are represented in only one of the at least one captured image or the received point cloud information; determine whether a vantage point difference between the first location of the camera and the second location of the LIDAR system accounts for the one or more detected objects being represented in only one of the at least one captured image or the received point cloud information; if the vantage point difference does not account for the one or more detected objects being represented in only one of the at least one captured image or the received point cloud information, then cause at least one remedial action to be taken; and if the vantage point difference does account for the one or more detected objects being represented in only one of the at least one captured image or the received point cloud information, then determine at least one navigational action to be taken by the host vehicle based on the one or more detected objects and cause the at least one navigational action to be taken by the host vehicle.