# US 12,214,717 B2 · Vehicular multi-sensor system using a camera and lidar sensor to detect objects > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,214,717 B2 - Document ID: US12214717B2 - Status: Granted - Kind: B2 - Application number: 18/468,776 - Filed: 2023-09-18 - Published: 2024-01-04 - Granted: 2025-02-04 - Earliest claimed: 2015-04-03 - Assignees: Magna Electronics Inc. - Applicants: Magna Electronics Inc. - Inventors: Martin Solar; Christopher L Van Dan Elzen - Web page: https://patentgrep.com/patent/US12214717B2 ## CPC - B60Q 1/143: {combined with another condition, e.g. using vehicle recognition from camera images or activation of wipers} - B60Q 1/0023: {the element being a sensor, e.g. distance sensor, camera} - B60R 11/04: Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle - B60W 30/14: {Adaptive} cruise control - G08G 1/161: {Decentralised systems, e.g. inter-vehicle communication} - G08G 1/163: {involving continuous checking} - G08G 1/164: {Centralised systems, e.g. external to vehicles} - G08G 1/166: {for active traffic, e.g. moving vehicles, pedestrians, bikes} - B60Q 2300/132: Pitch - B60Q 2300/41: preceding vehicle - B60Q 2300/42: oncoming vehicle - B60Q 2300/45: Special conditions, e.g. pedestrians, road signs or potential dangers - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2420/408: Radar; Laser, e.g. lidar - B60W 2556/65: Data transmitted between vehicles - B60Q: ARRANGEMENT OF SIGNALLING OR LIGHTING DEVICES, THE MOUNTING OR SUPPORTING THEREOF OR CIRCUITS THEREFOR, FOR VEHICLES IN GENERAL - B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93}) ## Abstract A vehicular multi-sensor system includes a plurality of sensors that include at least a forward-viewing camera and a forward-sensing 3D point-cloud LIDAR sensor. The forward-viewing camera views (i) a traffic lane of a multi-lane road being traveled along by the equipped vehicle and (ii) another traffic lane of the multi-lane road, and the field of sensing of the forward-sensing 3D point-cloud LIDAR sensor at least encompasses the other traffic lane of the multi-lane road. Image data captured by the forward-viewing camera is transferred to and is processed at an electronic control unit (ECU). 3D point-cloud LIDAR data captured by the forward-sensing 3D point-cloud LIDAR sensor is transferred to and processed at the ECU. Responsive at least in part to processing at the ECU of 3D point-cloud LIDAR data captured by the forward-sensing 3D point-cloud LIDAR sensor, a traffic participant present forward of the equipped vehicle is detected. ## Claims ### Claim 1 A vehicular multi-sensor system, said vehicular multi-sensor system comprising: a plurality of sensors disposed at a vehicle equipped with said vehicular multi-sensor system; wherein said plurality of sensors at least comprises (i) a forward-viewing camera, said forward-viewing camera disposed behind a windshield of the equipped vehicle and viewing forward of the equipped vehicle through the windshield and (ii) a forward-sensing 3D point-cloud LIDAR sensor having a field of sensing forward of the equipped vehicle; wherein said forward-viewing camera is operable to capture image data; wherein said forward-sensing 3D point-cloud LIDAR sensor is operable to capture 3D point-cloud LIDAR data; wherein said forward-viewing camera comprises a photosensor array having at least one million photosensor elements arranged in a two-dimensional array of columns and rows; wherein said forward-viewing camera views (i) a traffic lane of a multi-lane road being traveled along by the equipped vehicle and (ii) another traffic lane of the multi-lane road; wherein the field of sensing of said forward-sensing 3D point-cloud LIDAR sensor at least encompasses the other traffic lane of the multi-lane road; an electronic control unit (ECU) comprising electronic circuitry; wherein image data captured by said forward-viewing camera is transferred from said forward-viewing camera to the ECU and is processed at the ECU; wherein 3D point-cloud LIDAR data captured by said forward-sensing 3D point-cloud LIDAR sensor is transferred from said forward-sensing 3D point-cloud LIDAR sensor to the ECU and is processed at the ECU; wherein, responsive at least in part to processing at the ECU of 3D point-cloud LIDAR data captured by said forward-sensing 3D point-cloud LIDAR sensor, a traffic participant present forward of the equipped vehicle on the other traffic lane of the multi-lane road is detected; wherein image data captured by said forward-viewing camera is processed at the ECU for a headlamp control system of the equipped vehicle and for at least one selected from the group consisting of (i) a lane departure warning system of the equipped vehicle, (ii) a traffic sign recognition system of the equipped vehicle and (iii) an adaptive cruise control system of the equipped vehicle; and wherein the headlamp control system of the equipped vehicle adjusts a headlamp of the equipped vehicle based at least in part on detection of the traffic participant responsive at least in part to processing at the ECU of 3D point-cloud LIDAR data captured by said forward-sensing 3D point-cloud LIDAR sensor. ### Claim 2 (depends on 1) The vehicular multi-sensor system of claim 1, wherein the traffic participant present forward of the equipped vehicle on the other traffic lane of the multi-lane road comprises another vehicle present forward of the equipped vehicle. ### Claim 3 (depends on 2) The vehicular multi-sensor system of claim 2, wherein the other vehicle present forward of the equipped vehicle on the other traffic lane of the multi-lane road is approaching the equipped vehicle and is traveling along the multi-lane road being traveled along by the equipped vehicle. ### Claim 4 (depends on 3) The vehicular multi-sensor system of claim 3, wherein the other vehicle present forward of the equipped vehicle on the other traffic lane of the multi-lane road that is approaching the equipped vehicle is detected via processing at the ECU of image data captured by said forward-viewing camera. ### Claim 5 (depends on 1) The vehicular multi-sensor system of claim 1, wherein, responsive at least in part to processing at the ECU of 3D point-cloud LIDAR data captured by said forward-sensing 3D point-cloud LIDAR sensor, another traffic participant present forward of the equipped vehicle on another road that is different from the multi-lane road being traveled along by the equipped vehicle is detected. ### Claim 6 (depends on 5) The vehicular multi-sensor system of claim 5, wherein the other vehicle traffic participant on the other road that is different from the multi-lane road being traveled along by the equipped vehicle is detected via processing at the ECU of image data captured by said forward-viewing camera. ### Claim 7 (depends on 5) The vehicular multi-sensor system of claim 5, wherein the other traffic participant present forward of the equipped vehicle comprises another vehicle that is not traveling along the multi-lane road that is being traveled along by the equipped vehicle. ### Claim 8 (depends on 7) The vehicular multi-sensor system of claim 7, wherein the other vehicle that is not traveling along the multi-lane road being traveled along by the equipped vehicle is approaching a road intersection present forward of the equipped vehicle. ### Claim 9 (depends on 8) The vehicular multi-sensor system of claim 8, wherein the other vehicle that is not traveling along the multi-lane road being traveled along by the equipped vehicle and that is approaching the road intersection present forward of the equipped vehicle is detected via processing at the ECU of image data captured by said forward-viewing camera. ### Claim 10 (depends on 1) The vehicular multi-sensor system of claim 1, wherein the traffic participant present forward of the equipped vehicle on the other traffic lane of the multi-lane road comprises a pedestrian. ### Claim 11 (depends on 10) The vehicular multi-sensor system of claim 10, wherein image data captured by said forward-viewing camera is processed at the ECU for the adaptive cruise control system of the equipped vehicle. ### Claim 12 (depends on 1) The vehicular multi-sensor system of claim 1, wherein the traffic participant present forward of the equipped vehicle on the other traffic lane of the multi-lane road comprises a motorcyclist. ### Claim 13 (depends on 1) The vehicular multi-sensor system of claim 1, wherein the traffic participant present forward of the equipped vehicle on the other traffic of the multi-lane road comprises a bicyclist. ### Claim 14 (depends on 13) The vehicular multi-sensor system of claim 13, wherein image data captured by said forward-viewing camera is processed at the ECU for the lane departure warning system of the equipped vehicle. ### Claim 15 (depends on 14) The vehicular multi-sensor system of claim 14, wherein image data captured by said forward-viewing camera is processed at the ECU for the traffic sign recognition system of the equipped vehicle. ### Claim 16 (depends on 1) The vehicular multi-sensor system of claim 1, wherein said forward-sensing 3D point-cloud LIDAR sensor comprises a solid-state 3D point-cloud LIDAR sensor. ### Claim 17 (depends on 1) The vehicular multi-sensor system of claim 1, wherein image data captured by said forward-viewing camera is processed at the ECU by an image processor that utilizes object detection software for processing image data captured by said forward-viewing camera. ### Claim 18 (depends on 17) The vehicular multi-sensor system of claim 17, wherein said image processor comprises an image processing chip. ### Claim 19 (depends on 1) The vehicular multi-sensor system of claim 1, wherein the ECU determines, via processing at the ECU (i) of image data captured by said forward-viewing camera and (ii) of 3D point-cloud LIDAR data captured by said forward-sensing 3D point-cloud LIDAR sensor, that the detected traffic participant is not yet viewed by said forward-viewing camera. ### Claim 20 (depends on 1) The vehicular multi-sensor system of claim 1, wherein said plurality of sensors comprises a forward-sensing night vision camera, and wherein said forward-sensing night vision camera is operable to capture image data, and wherein image data captured by said forward-sensing night vision camera is transferred from said forward-sensing night vision camera to the ECU and is processed at the ECU. ### Claim 21 A vehicular multi-sensor system, said vehicular multi-sensor system comprising: a plurality of sensors disposed at a vehicle equipped with said vehicular multi-sensor system; wherein said plurality of sensors at least comprises (i) a forward-viewing camera, said forward-viewing camera disposed behind a windshield of the equipped vehicle and viewing forward of the equipped vehicle through the windshield, (ii) a forward-sensing 3D point-cloud LIDAR sensor having a field of sensing forward of the equipped vehicle and (iii) a forward-sensing night vision camera; wherein said forward-viewing camera is operable to capture image data; wherein said forward-sensing 3D point-cloud LIDAR sensor is operable to capture 3D point-cloud LIDAR data; wherein said forward-sensing night vision camera is operable to capture image data; wherein said forward-viewing camera comprises a photosensor array having at least one million photosensor elements arranged in a two-dimensional array of columns and rows; wherein said forward-viewing camera views (i) a traffic lane of a multi-lane road being traveled along by the equipped vehicle and (ii) another traffic lane of the multi-lane road; wherein the field of sensing of said forward-sensing 3D point-cloud LIDAR sensor at least encompasses the other traffic lane of the multi-lane road; an electronic control unit (ECU) comprising electronic circuitry; wherein image data captured by said forward-viewing camera is transferred from said forward-viewing camera to the ECU and is processed at the ECU; wherein 3D point-cloud LIDAR data captured by said forward-sensing 3D point-cloud LIDAR sensor is transferred from said forward-sensing 3D point-cloud LIDAR sensor to the ECU and is processed at the ECU; wherein image data captured by said forward-sensing night vision camera is transferred from said forward-sensing night vision camera to the ECU and is processed at the ECU; wherein, responsive at least in part to processing at the ECU of 3D point-cloud LIDAR data captured by said forward-sensing 3D point-cloud LIDAR sensor and of at least one selected from the group consisting of (i) image data captured by said forward-viewing camera and (ii) image data captured by said forward-sensing night vision camera, a traffic participant present forward of the equipped vehicle on the other traffic lane of the multi-lane road is detected; wherein the traffic participant present forward of the equipped vehicle on the other traffic lane of the multi-lane road comprises a pedestrian; wherein image data captured by said forward-viewing camera is processed at the ECU for a headlamp control system of the equipped vehicle and for at least one selected from the group consisting of (i) a lane departure warning system of the equipped vehicle, (ii) a traffic sign recognition system of the equipped vehicle and (iii) an adaptive cruise control system of the equipped vehicle; and wherein the headlamp control system of the equipped vehicle adjusts a headlamp of the equipped vehicle based at least in part on detection of the traffic participant responsive at least in part to processing at the ECU of 3D point-cloud LIDAR data captured by said forward-sensing 3D point-cloud LIDAR sensor. ### Claim 22 (depends on 21) The vehicular multi-sensor system of claim 21, wherein the traffic participant present forward of the equipped vehicle comprises another vehicle on the multi-lane road being traveled along by the equipped vehicle. ### Claim 23 (depends on 22) The vehicular multi-sensor system of claim 22, wherein the traffic participant present forward of the equipped vehicle on the other traffic lane of the multi-lane road is approaching the equipped vehicle. ### Claim 24 (depends on 22) The vehicular multi-sensor system of claim 22, wherein image data captured by said forward-viewing camera is processed at the ECU for the lane departure warning system of the equipped vehicle. ### Claim 25 (depends on 24) The vehicular multi-sensor system of claim 24, wherein image data captured by said forward-viewing camera is processed at the ECU for the traffic sign recognition system of the equipped vehicle. ### Claim 26 (depends on 22) The vehicular multi-sensor system of claim 22, wherein image data captured by said forward-viewing camera is processed at the ECU for the adaptive cruise control system of the equipped vehicle. ### Claim 27 (depends on 22) The vehicular multi-sensor system of claim 22, wherein said forward-sensing 3D point-cloud LIDAR sensor comprises a solid-state 3D point-cloud LIDAR sensor. ### Claim 28 (depends on 22) The vehicular multi-sensor system of claim 22, wherein image data captured by said forward-viewing camera is processed at the ECU by an image processor that utilizes object detection software for processing image data captured by said forward-viewing camera. ### Claim 29 (depends on 28) The vehicular multi-sensor system of claim 28, wherein said image processor comprises an image processing chip. ### Claim 30 (depends on 21) The vehicular multi-sensor system of claim 21, wherein, responsive at least in part to processing at the ECU of 3D point-cloud LIDAR data captured by said forward-sensing 3D point-cloud LIDAR sensor, another traffic participant present forward of the equipped vehicle is not on the multi-lane road that is being traveled along by the equipped vehicle is detected. ### Claim 31 (depends on 21) The vehicular multi-sensor system of claim 21, wherein the ECU determines, via processing at the ECU (i) of image data captured by said forward-viewing camera and (ii) of 3D point-cloud LIDAR data captured by said forward-sensing 3D point-cloud LIDAR sensor, that the detected traffic participant is not yet viewed by said forward-viewing camera. ### Claim 32 A vehicular multi-sensor system, said vehicular multi-sensor system comprising: a plurality of sensors disposed at a vehicle equipped with said vehicular multi-sensor system; wherein said plurality of sensors at least comprises (i) a forward-viewing camera, said forward-viewing camera disposed behind a windshield of the equipped vehicle and viewing forward of the equipped vehicle through the windshield and (ii) a forward-sensing 3D point-cloud LIDAR sensor having a field of sensing forward of the equipped vehicle; wherein said forward-viewing camera is operable to capture image data; wherein said forward-sensing 3D point-cloud LIDAR sensor is operable to capture 3D point-cloud LIDAR data; wherein said forward-viewing camera comprises a photosensor array having at least one million photosensor elements arranged in a two-dimensional array of columns and rows; wherein said forward-viewing camera views (i) a traffic lane of a multi-lane road being traveled along by the equipped vehicle and (ii) another traffic lane of the multi-lane road; wherein the field of sensing of said forward-sensing 3D point-cloud LIDAR sensor at least encompasses the other traffic lane of the multi-lane road; an electronic control unit (ECU) comprising electronic circuitry; wherein image data captured by said forward-viewing camera is transferred from said forward-viewing camera to the ECU and is processed at the ECU; wherein 3D point-cloud LIDAR data captured by said forward-sensing 3D point-cloud LIDAR sensor is transferred from said forward-sensing 3D point-cloud LIDAR sensor to the ECU and is processed at the ECU; wherein, responsive at least in part to processing at the ECU (i) of image data captured by said forward-viewing camera and (ii) of 3D point-cloud LIDAR data captured by said forward-sensing 3D point-cloud LIDAR sensor, a traffic participant present forward of the equipped vehicle on the other traffic lane of the multi-lane road is detected; wherein the traffic participant present forward of the equipped vehicle on the other traffic lane of the multi-lane road comprises another vehicle present forward of the equipped vehicle; wherein image data captured by said forward-viewing camera is processed at the ECU for a headlamp control system of the equipped vehicle and for at least one selected from the group consisting of (i) a lane departure warning system of the equipped vehicle, (ii) a traffic sign recognition system of the equipped vehicle and (iii) an adaptive cruise control system of the equipped vehicle; and wherein the headlamp control system of the equipped vehicle adjusts a headlamp of the equipped vehicle based at least in part on detection of the traffic participant responsive at least in part to processing at the ECU of 3D point-cloud LIDAR data captured by said forward-sensing 3D point-cloud LIDAR sensor. ### Claim 33 (depends on 32) The vehicular multi-sensor system of claim 32, wherein image data captured by said forward-viewing camera is processed at the ECU by an image processor that utilizes object detection software for processing image data captured by said forward-viewing camera, and wherein the traffic participant present forward of the equipped vehicle on the other traffic lane of the multi-lane road is approaching the equipped vehicle and is traveling along the multi-lane road being traveled along by the equipped vehicle. ### Claim 34 (depends on 32) The vehicular multi-sensor system of claim 32, wherein, responsive at least in part to processing at the ECU of 3D point-cloud LIDAR data captured by said forward-sensing 3D point-cloud LIDAR sensor, another traffic participant present forward of the equipped vehicle is traveling along another road that is different from the multi-lane road being traveled along by the equipped vehicle is detected. ### Claim 35 (depends on 32) The vehicular multi-sensor system of claim 32, wherein, wherein responsive at least in part to processing at the ECU of 3D point-cloud LIDAR data captured by said forward-sensing 3D point-cloud LIDAR sensor, another traffic participant present forward of the equipped vehicle is not traveling along the multi-lane road that is being traveled along by the equipped vehicle is detected. ### Claim 36 (depends on 35) The vehicular multi-sensor system of claim 35, wherein the other traffic participant that is not traveling along the multi-lane road being traveled along by the equipped vehicle is approaching a road intersection present forward of the equipped vehicle. ### Claim 37 (depends on 32) The vehicular multi-sensor system of claim 32, wherein the traffic participant present forward of the equipped vehicle is approaching the equipped vehicle, and wherein image data captured by said forward-viewing camera is processed at the ECU for the lane departure warning system of the equipped vehicle. ### Claim 38 (depends on 37) The vehicular multi-sensor system of claim 37, wherein image data captured by said forward-viewing camera is processed at the ECU for the traffic sign recognition system of the equipped vehicle. ### Claim 39 (depends on 32) The vehicular multi-sensor system of claim 32, wherein image data captured by said forward-viewing camera is processed at the ECU for the adaptive cruise control system of the equipped vehicle. ### Claim 40 (depends on 39) The vehicular multi-sensor system of claim 39, wherein said forward-sensing 3D point-cloud LIDAR sensor comprises a solid-state 3D point-cloud LIDAR sensor. ### Claim 41 (depends on 32) The vehicular multi-sensor system of claim 32, wherein image data captured by said forward-viewing camera is processed at the ECU by an image processor that utilizes object detection software for processing image data captured by said forward-viewing camera. ### Claim 42 (depends on 41) The vehicular multi-sensor system of claim 41, wherein said image processor comprises an image processing chip. ### Claim 43 (depends on 42) The vehicular multi-sensor system of claim 42, wherein the ECU determines, via processing at the ECU (i) of image data captured by said forward-viewing camera and (ii) of 3D point-cloud LIDAR data captured by said forward-sensing 3D point-cloud LIDAR sensor, that the detected traffic participant is not yet viewed by said forward-viewing camera. ### Claim 44 (depends on 32) The vehicular multi-sensor system of claim 32, wherein said plurality of sensors comprises a forward-sensing night vision camera, and wherein said forward-sensing night vision camera is operable to capture image data, and wherein image data captured by said forward-sensing night vision camera is transferred from said forward-sensing night vision camera to the ECU and is processed at the ECU.