# US 11,908,166 B2 · Vehicular imaging system with misalignment correction of camera > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,166 B2 - Document ID: US11908166B2 - Status: Granted - Kind: B2 - Application number: 17/662,447 - Filed: 2022-05-09 - Published: 2022-08-18 - Granted: 2024-02-20 - Earliest claimed: 2007-08-17 - Assignees: Magna Electronics Inc. - Applicants: Magna Electronics Inc. - Inventors: Yuesheng Lu; Michael J. Higgins-Luthman; Antony V. Jeyaraj; Manoj R. Phirke - Web page: https://patentgrep.com/patent/US11908166B2 ## CPC - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - B60Q 1/085: {due to special conditions, e.g. adverse weather, type of road, badly illuminated road signs or potential dangers (B60Q1/10, B60Q1/12, B60Q1/1423 take precedence)} - B60Q 1/1423: {Automatic dimming circuits, i.e. switching between high beam and low beam due to change of ambient light or light level in road traffic} - B60R 1/24: in front of the vehicle - B60R 1/28: with an adjustable field of view - G06V 20/584: {of vehicle lights or traffic lights} - G06V 20/588: {Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road} - B60Q 2300/41: preceding vehicle - B60Q 2300/42: oncoming vehicle - G06T 2207/10016: Video; Image sequence - G06T 2207/30252: Vehicle exterior; Vicinity of vehicle - G06T 2207/30256: Lane; Road marking - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - B60Q: ARRANGEMENT OF SIGNALLING OR LIGHTING DEVICES, THE MOUNTING OR SUPPORTING THEREOF OR CIRCUITS THEREFOR, FOR VEHICLES IN GENERAL - B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A vehicular imaging system includes a camera disposed behind a windshield of a vehicle and viewing through a portion of the windshield. Image data captured by the camera is provided to a control. The control receives, via a communication bus of the vehicle, at least one selected from the group consisting of (i) vehicle pitch information relating to pitch of the vehicle, (ii) vehicle yaw information relating to yaw of the vehicle and (iii) vehicle steering information relating to steering of the vehicle. The system automatically corrects for misalignment of the camera. Image data captured by the camera is processed at the control for a lane departure warning system of the vehicle and for at least one selected from the group consisting of (i) an automatic headlamp control system of the vehicle, (ii) a collision avoidance system of the vehicle and (iii) an adaptive front lighting system of the vehicle. ## Claims ### Claim 1 A vehicular imaging system, said vehicular imaging system comprising: a camera comprising a CMOS imaging sensor; said imaging sensor comprising a photosensor array having a plurality of photosensor elements; wherein said camera is disposed in a vehicle equipped with said vehicular imaging system, and wherein said camera is disposed behind a windshield of the equipped vehicle and views through a portion of the windshield; wherein said camera views a scene forward of the equipped vehicle; wherein the portion of the windshield comprises a portion cleaned by a windshield wiper of the equipped vehicle; wherein said camera is operable to capture frames of image data at an image frame capture rate that is between 5 times per second and 120 times per second; wherein image data captured by said camera is provided to a control, said control comprising an image processor; wherein said control receives, via a communication bus of the equipped vehicle, at least one selected from the group consisting of (i) vehicle pitch information relating to pitch of the equipped vehicle, (ii) vehicle yaw information relating to yaw of the equipped vehicle and (iii) vehicle steering information relating to steering of the equipped vehicle; wherein said vehicular imaging system automatically corrects for misalignment of said camera disposed behind the windshield of the equipped vehicle and viewing through the portion of the windshield; and wherein image data captured by said camera is processed at said control for a lane departure warning system of the equipped vehicle and for at least one selected from the group consisting of (i) an automatic headlamp control system of the equipped vehicle, (ii) a collision avoidance system of the equipped vehicle and (iii) an adaptive front lighting system of the equipped vehicle. ### Claim 2 (depends on 1) The vehicular imaging system of claim 1, wherein said vehicular imaging system automatically corrects for misalignment of said camera of up to +/−4 degrees from an aligned condition of said camera. ### Claim 3 (depends on 2) The vehicular imaging system of claim 2, wherein said control corrects a misalignment of said camera that occurs due to use of the equipped vehicle. ### Claim 4 (depends on 3) The vehicular imaging system of claim 3, wherein the misalignment that occurs due to use of the equipped vehicle arises from at least one selected from the group consisting of (i) a change in pitch of the equipped vehicle, (ii) a change in tilt of the equipped vehicle and (iii) a change in yaw of the equipped vehicle. ### Claim 5 (depends on 3) The vehicular imaging system of claim 3, wherein the misalignment arises from a load condition of the equipped vehicle. ### Claim 6 (depends on 1) The vehicular imaging system of claim 1, wherein said control is operable to process a reduced set of image data more than other image data. ### Claim 7 (depends on 1) The vehicular imaging system of claim 1, wherein said control processes captured image data at a higher resolution level of processing of captured image data and at a lower resolution level of processing of captured image data. ### Claim 8 (depends on 7) The vehicular imaging system of claim 7, wherein said control processes captured image data at the lower resolution level responsive to distance between the equipped vehicle and an object viewed by said camera being below a threshold level for the object, and wherein said control processes captured image data at the higher resolution level responsive to distance between the equipped vehicle and the object viewed by said camera being above the threshold level for the object. ### Claim 9 (depends on 1) The vehicular imaging system of claim 1, wherein said vehicular imaging system detects when at least a portion of said photosensor array of said imaging sensor of said camera is blocked. ### Claim 10 (depends on 9) The vehicular imaging system of claim 9, wherein said vehicular imaging system determines that said photosensor array of said camera is totally blocked when, over a plurality of frames of image data captured by said camera, processing of image data at said control determines that a count of bright pixels remains below a threshold. ### Claim 11 (depends on 9) The vehicular imaging system of claim 9, wherein said vehicular imaging system determines that said photosensor array of said camera is partially blocked via said control performing region-based image processing to take into account intensity variations in different regions of said photosensor array of said camera. ### Claim 12 (depends on 1) The vehicular imaging system of claim 1, wherein said vehicular imaging system automatically detects a driving side of a road being traveled along by the equipped vehicle. ### Claim 13 (depends on 12) The vehicular imaging system of claim 12, wherein image processing at said control of image data captured by said camera identifies a taillight of another vehicle that is 200 meters in front of the equipped vehicle. ### Claim 14 (depends on 1) The vehicular imaging system of claim 1, wherein said vehicular imaging system is operable to encode signals into a light output of a light of the equipped vehicle to allow an imaging system of another vehicle to detect an encoded signal. ### Claim 15 (depends on 1) The vehicular imaging system of claim 1, wherein said vehicular imaging system is operable to detect and ameliorate a bad photosensor element of said plurality of photosensor elements of said imaging sensor of said camera. ### Claim 16 (depends on 1) The vehicular imaging system of claim 1, wherein a reduced image data set of image data captured by said camera is processed at said control, the reduced image data set being representative of a portion of the captured image data as captured by a particular grouping of said photosensor elements of said imaging sensor of said camera. ### Claim 17 (depends on 1) The vehicular imaging system of claim 1, wherein the automatic correction for misalignment of said camera mounted at the equipped vehicle comprises image processing at said control of image data captured by said camera determining where converging road features of a road along which the equipped vehicle is traveling would converge. ### Claim 18 (depends on 1) The vehicular imaging system of claim 1, wherein the automatic correction for misalignment of said camera comprises comparing an imaged location of an object present in a field of view of said imaging sensor to an expected location of the object. ### Claim 19 (depends on 1) The vehicular imaging system of claim 1, wherein the automatic correction for misalignment of said camera mounted at the equipped vehicle comprises image processing at said control of image data captured by said camera comparing an imaged location of an object viewed by said camera to an expected location of the object, wherein the imaged location is determined by image processing at said control of image data captured said camera and wherein the expected location is determined responsive to vehicle data carried to said control via said communication bus. ### Claim 20 A vehicular imaging system, said vehicular imaging system comprising: a camera comprising a CMOS imaging sensor; said imaging sensor comprising a photosensor array having a plurality of photosensor elements; wherein said camera is disposed in a vehicle equipped with said vehicular imaging system, and wherein said camera is disposed behind a windshield of the equipped vehicle and views through a portion of the windshield; wherein said camera views a scene forward of the equipped vehicle; wherein the portion of the windshield comprises a portion cleaned by a windshield wiper of the equipped vehicle; wherein said camera is operable to capture frames of image data at an image frame capture rate that is between 5 times per second and 120 times per second; wherein image data captured by said camera is provided to a control, said control comprising an image processor; wherein said control receives, via a communication bus of the equipped vehicle, at least one selected from the group consisting of (i) vehicle pitch information relating to pitch of the equipped vehicle, (ii) vehicle yaw information relating to yaw of the equipped vehicle and (iii) vehicle steering information relating to steering of the equipped vehicle; wherein said vehicular imaging system automatically corrects for misalignment of said camera disposed behind the windshield of the equipped vehicle and viewing through the portion of the windshield; wherein said vehicular imaging system automatically corrects for misalignment of said camera of up to +/−4 degrees from an aligned condition of said camera; wherein the automatic correction for misalignment of said camera mounted at the equipped vehicle comprises image processing at said control of image data captured by said camera determining where converging road features of a road along which the equipped vehicle is traveling would converge; and wherein image data captured by said camera is processed at said control for a lane departure warning system of the equipped vehicle and for at least one selected from the group consisting of (i) an automatic headlamp control system of the equipped vehicle, (ii) a collision avoidance system of the equipped vehicle and (iii) an adaptive front lighting system of the equipped vehicle. ### Claim 21 (depends on 20) The vehicular imaging system of claim 20, wherein image processing at said control of image data captured by said camera identifies a taillight of another vehicle that is 200 meters in front of the equipped vehicle. ### Claim 22 (depends on 21) The vehicular imaging system of claim 21, wherein said control corrects a misalignment of said camera that occurs due to use of the equipped vehicle. ### Claim 23 (depends on 22) The vehicular imaging system of claim 22, wherein the misalignment that occurs due to use of the equipped vehicle arises from at least one selected from the group consisting of (i) a change in pitch of the equipped vehicle, (ii) a change in tilt of the equipped vehicle and (iii) a change in yaw of the equipped vehicle. ### Claim 24 (depends on 20) The vehicular imaging system of claim 20, wherein said control is operable to process a reduced set of image data more than other image data. ### Claim 25 (depends on 24) The vehicular imaging system of claim 24, wherein said control processes captured image data at a higher resolution level of processing of captured image data and at a lower resolution level of processing of captured image data. ### Claim 26 (depends on 25) The vehicular imaging system of claim 25, wherein said control processes captured image data at the lower resolution level responsive to distance between the equipped vehicle and an object viewed by said camera being below a threshold level for the object, and wherein said control processes captured image data at the higher resolution level responsive to distance between the equipped vehicle and the object viewed by said camera being above the threshold level for the object. ### Claim 27 (depends on 20) The vehicular imaging system of claim 20, wherein said vehicular imaging system detects when at least a portion of said photosensor array of said imaging sensor of said camera is blocked. ### Claim 28 (depends on 27) The vehicular imaging system of claim 27, wherein said vehicular imaging system determines that said photosensor array of said camera is totally blocked when, over a plurality of frames of image data captured by said camera, processing of image data at said control determines that a count of bright pixels remains below a threshold. ### Claim 29 (depends on 27) The vehicular imaging system of claim 27, wherein said vehicular imaging system determines that said photosensor array of said camera is partially blocked via said control performing region-based image processing to take into account intensity variations in different regions of said photosensor array of said camera. ### Claim 30 (depends on 20) The vehicular imaging system of claim 20, wherein said vehicular imaging system automatically detects a driving side of the road being traveled along by the equipped vehicle. ### Claim 31 (depends on 30) The vehicular imaging system of claim 30, wherein a reduced image data set of image data captured by said camera is processed at said control, the reduced image data set being representative of a portion of the captured image data as captured by a particular grouping of said photosensor elements of said imaging sensor of said camera. ### Claim 32 A vehicular imaging system, said vehicular imaging system comprising: a camera comprising a CMOS imaging sensor; said imaging sensor comprising a photosensor array having a plurality of photosensor elements; wherein said camera is disposed in a vehicle equipped with said vehicular imaging system, and wherein said camera is disposed behind a windshield of the equipped vehicle and views through a portion of the windshield; wherein said camera views a scene forward of the equipped vehicle; wherein the portion of the windshield comprises a portion cleaned by a windshield wiper of the equipped vehicle; wherein said camera is operable to capture frames of image data at an image frame capture rate that is between 5 times per second and 120 times per second; wherein image data captured by said camera is provided to a control, said control comprising an image processor; wherein said control receives, via a communication bus of the equipped vehicle, at least one selected from the group consisting of (i) vehicle pitch information relating to pitch of the equipped vehicle, (ii) vehicle yaw information relating to yaw of the equipped vehicle and (iii) vehicle steering information relating to steering of the equipped vehicle; wherein said vehicular imaging system automatically corrects for misalignment of said camera disposed behind the windshield of the equipped vehicle and viewing through the portion of the windshield; wherein said vehicular imaging system automatically corrects for misalignment of said camera of up to +/−4 degrees from an aligned condition of said camera; wherein the automatic correction for misalignment of said camera mounted at the equipped vehicle comprises image processing at said control of image data captured by said camera comparing an imaged location of an object viewed by said camera to an expected location of the object; wherein the imaged location is determined by image processing at said control of image data captured said camera; wherein the expected location is determined responsive to vehicle data carried to said control via said communication bus; and wherein image data captured by said camera is processed at said control for a lane departure warning system of the equipped vehicle and for at least one selected from the group consisting of (i) an automatic headlamp control system of the equipped vehicle, (ii) a collision avoidance system of the equipped vehicle and (iii) an adaptive front lighting system of the equipped vehicle. ### Claim 33 (depends on 32) The vehicular imaging system of claim 32, wherein image processing at said control of image data captured by said camera identifies a taillight of another vehicle that is 200 meters in front of the equipped vehicle. ### Claim 34 (depends on 33) The vehicular imaging system of claim 33, wherein said vehicular imaging system automatically detects a driving side of a road being traveled along by the equipped vehicle. ### Claim 35 (depends on 34) The vehicular imaging system of claim 34, wherein a reduced image data set of image data captured by said camera is processed at said control, the reduced image data set being representative of a portion of the captured image data as captured by a particular grouping of said photosensor elements of said imaging sensor of said camera. ### Claim 36 (depends on 33) The vehicular imaging system of claim 33, wherein said control is operable to process a reduced set of image data more than other image data. ### Claim 37 (depends on 32) The vehicular imaging system of claim 32, wherein said control corrects a misalignment of said camera that occurs due to use of the equipped vehicle. ### Claim 38 (depends on 37) The vehicular imaging system of claim 37, wherein the misalignment that occurs due to use of the equipped vehicle arises from at least one selected from the group consisting of (i) a change in pitch of the equipped vehicle, (ii) a change in tilt of the equipped vehicle and (iii) a change in yaw of the equipped vehicle. ### Claim 39 (depends on 32) The vehicular imaging system of claim 32, wherein said control processes captured image data at a higher resolution level of processing of captured image data and at a lower resolution level of processing of captured image data. ### Claim 40 (depends on 39) The vehicular imaging system of claim 39, wherein said control processes captured image data at the lower resolution level responsive to distance between the equipped vehicle and an object viewed by said camera being below a threshold level for the object, and wherein said control processes captured image data at the higher resolution level responsive to distance between the equipped vehicle and the object viewed by said camera being above the threshold level for the object. ### Claim 41 (depends on 32) The vehicular imaging system of claim 32, wherein said vehicular imaging system detects when at least a portion of said photosensor array of said imaging sensor of said camera is blocked. ### Claim 42 (depends on 41) The vehicular imaging system of claim 41, wherein said vehicular imaging system determines that said photosensor array of said camera is totally blocked when, over a plurality of frames of image data captured by said camera, processing of image data at said control determines that a count of bright pixels remains below a threshold. ### Claim 43 (depends on 41) The vehicular imaging system of claim 41, wherein said vehicular imaging system determines that said photosensor array of said camera is partially blocked via said control performing region-based image processing to take into account intensity variations in different regions of said photosensor array of said camera.