# US 12,602,934 B2 · Vehicular driving assist system with traffic light recognition > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,934 B2 - Document ID: US12602934B2 - Status: Granted - Kind: B2 - Application number: 18/754,270 - Filed: 2024-06-26 - Published: 2025-01-02 - Granted: 2026-04-14 - Earliest claimed: 2023-06-29 - Assignees: Magna Electronics Inc. - Applicants: Magna Electronics Inc. - Inventors: Dhruv Mathur; Tejas Murlidhar Varunjikar; Nidhip Naik - Web page: https://patentgrep.com/patent/US12602934B2 ## CPC - G06V 20/584: {of vehicle lights or traffic lights} - B60Q 9/00: Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 {, e.g. haptic signalling} - B60R 11/04: Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B60Q: ARRANGEMENT OF SIGNALLING OR LIGHTING DEVICES, THE MOUNTING OR SUPPORTING THEREOF OR CIRCUITS THEREFOR, FOR VEHICLES IN GENERAL - B60R: VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR ## Abstract A vehicular vision system includes a camera disposed at a vehicle and viewing at least forward of the vehicle. The vehicular vision system, as the vehicle travels along a traffic lane of a road and via processing of image data captured by the camera and based on map data, determines presence of a plurality of traffic lights at an intersection and identifies a target traffic light from the plurality of traffic lights that is associated with the traffic lane along which the equipped vehicle is traveling as the vehicle approaches the intersection. The vehicular vision system determines, via processing of image data captured by the camera, status of the identified target traffic light. The vehicular vision system, responsive to determining the status of the identified target traffic light, alerts a driver of the vehicle of the status of the relevant traffic light and/or controls speed of the vehicle. ## Claims ### Claim 1 A vehicular vision system, the vehicular vision system comprising: a camera disposed at a vehicle equipped with the vehicular vision system, the camera viewing at least forward of the equipped vehicle; wherein the camera is operable to capture image data; wherein the camera comprises a CMOS imaging array, and wherein the CMOS imaging array comprises at least one million photosensors arranged in rows and columns; an electronic control unit (ECU) comprising electronic circuitry and associated software; wherein image data captured by the camera is transferred to the ECU; wherein the electronic circuitry of the ECU comprises at least one image processor; wherein the ECU is operable to process image data captured by the camera and transferred to the ECU; wherein, as the vehicle travels along a traffic lane of a road, the vehicular vision system, (i) via processing at the ECU of image data captured by the camera and (ii) based on map data, determines presence of a plurality of traffic lights at an intersection that the equipped vehicle is approaching, and wherein each traffic light of the plurality of traffic lights is associated with one or more traffic lanes of the road at the intersection; wherein the vehicular vision system, based on processing at the ECU of image data captured by the camera, determines a first relevancy score and a first confidence value associated with the first relevancy score; wherein the vehicular vision system, based on the map data, determines a curvature of the road; wherein the vehicular vision system, based on the determined curvature of the road, determines a second relevancy score and a second confidence value associated with the second relevancy score; wherein the vehicular vision system, based on the first relevancy score and the second relevancy score, determines an aggregated relevancy score by weighting the first relevancy score and the second relevancy score, wherein a higher weight is applied to the first relevancy score or the second relevancy score based on the first confidence value and the second confidence value; wherein the vehicular vision system, based at least in part on the aggregated relevancy score, identifies a target traffic light from the plurality of traffic lights that is associated with the traffic lane along which the equipped vehicle is traveling as the vehicle approaches the intersection; wherein the vehicular vision system determines a status of the identified target traffic light via processing at the ECU of image data captured by the camera; and wherein the vehicular vision system, responsive to determining the status of the identified target traffic light, performs at least one selected from the group consisting of (i) alerting a driver of the equipped vehicle of status of the target traffic light and (ii) controlling speed of the equipped vehicle. ### Claim 2 (depends on 1) The vehicular vision system of claim 1, wherein the vehicular vision system determines the first relevancy score based at least in part on (i) a longitudinal distance of the target traffic light from the equipped vehicle and (ii) a lateral distance of the target traffic light from the equipped vehicle. ### Claim 3 (depends on 1) The vehicular vision system of claim 1, wherein the vehicular vision system determines the first relevancy score based at least in part on vehicle dynamics data. ### Claim 4 (depends on 1) The vehicular vision system of claim 1, wherein the vehicular vision system determines the second relevancy score based at least in part on global positioning system data. ### Claim 5 (depends on 1) The vehicular vision system of claim 1, wherein the vehicular vision system determines the second relevancy score based at least in part on a longitudinal distance between the equipped vehicle and the intersection. ### Claim 6 (depends on 5) The vehicular vision system of claim 5, wherein the map data further comprises information regarding waypoints and curvature of the road associated with the traffic lane the equipped vehicle is traveling along. ### Claim 7 (depends on 1) The vehicular vision system of claim 1, wherein the vehicular vision system determines topology information for the traffic lane based on processing at the ECU of image data captured by the camera, and wherein the first relevancy score is further based on the topology information. ### Claim 8 (depends on 1) The vehicular vision system of claim 1, wherein the vehicular vision system determines, based on the aggregated relevancy score and vehicle dynamics data, a stopping distance for the equipped vehicle that will stop the vehicle prior to entering the intersection. ### Claim 9 (depends on 1) The vehicular vision system of claim 1, wherein the vehicular vision system determines, based on the aggregated relevancy score and vehicle dynamics data, an amount of time remaining until the equipped vehicle will enter the intersection. ### Claim 10 (depends on 9) The vehicular vision system of claim 9, wherein the vehicular vision system determines an amount of time remaining in which the equipped vehicle can enter the intersection before color of an illuminated indicator associated with the identified target traffic light changes, and wherein the vehicular vision system, responsive to the determined amount of time being less than a threshold time period, alerts the driver of the equipped vehicle. ### Claim 11 (depends on 1) The vehicular vision system of claim 1, wherein the alert comprises at least one selected from the group consisting of (i) a visual notification, (ii) an audible notification and (iii) a haptic notification. ### Claim 12 (depends on 11) The vehicular vision system of claim 11, wherein the alert comprises the visual notification, and wherein the visual notification includes a color of an illuminated indicator of the identified target traffic light. ### Claim 13 (depends on 1) The vehicular vision system of claim 1, wherein the vehicular vision system determines a color of an illuminated indicator associated with the identified target traffic light. ### Claim 14 (depends on 13) The vehicular vision system of claim 13, wherein the vehicular vision system determines a change in color of the illuminated indicator associated with the identified target traffic light, and wherein the vehicular vision system, responsive to the determined change in color of the illuminated indicator, alerts the driver of the equipped vehicle. ### Claim 15 (depends on 1) The vehicular vision system of claim 1, wherein the vehicular vision system ignores traffic lights of the plurality of traffic lights that are not identified to be the target traffic light. ### Claim 16 (depends on 1) The vehicular vision system of claim 1, wherein the status of the identified target traffic light comprises a status of an illuminated indicator of the identified target traffic light, and wherein the status of the illuminated indicator comprises at least one from the group consisting of (i) a color of the illuminated indicator and (ii) an arrow of the illuminated indicator. ### Claim 17 (depends on 1) The vehicular vision system of claim 1, wherein the identified target traffic light comprises one selected from the group consisting of (i) a left turn indicator for a left turn traffic lane of the road, (ii) a straight through indicator for a traffic lane that crosses the intersection and (iii) a right turn indicator for a right turn traffic lane of the road, and wherein other traffic lights of the plurality of traffic lights comprise at least one other selected from the group consisting of (i) the left turn indicator for a left turn traffic lane of the road, (ii) the straight through indicator for a traffic lane that crosses the intersection and (iii) the right turn indicator for a right turn traffic lane of the road. ### Claim 18 (depends on 1) The vehicular vision system of claim 1, wherein the vehicular vision system controls speed of the equipped vehicle via controlling braking of the equipped vehicle. ### Claim 19 A vehicular vision system, the vehicular vision system comprising: a camera disposed at a windshield of a vehicle equipped with the vehicular vision system, the camera viewing at least forward of the equipped vehicle through the windshield of the equipped vehicle; wherein the camera is operable to capture image data; wherein the camera comprises a CMOS imaging array, and wherein the CMOS imaging array comprises at least one million photosensors arranged in rows and columns; an electronic control unit (ECU) comprising electronic circuitry and associated software; wherein image data captured by the camera is transferred to the ECU; wherein the electronic circuitry of the ECU comprises at least one image processor; wherein the ECU is operable to process image data captured by the camera and transferred to the ECU; wherein, as the vehicle travels along a traffic lane of a road, the vehicular vision system, (i) via processing at the ECU of image data captured by the camera and (ii) based on map data, determines presence of a plurality of traffic lights at an intersection that the equipped vehicle is approaching, and wherein each traffic light of the plurality of traffic lights is associated with one or more traffic lanes of the road at the intersection; wherein the vehicular vision system determines a first relevancy score based on processing at the ECU of image data captured by the camera and a first confidence value associated with the first relevancy score; wherein the vehicular vision system, based on the map data, determines a curvature of the road; wherein the vehicular vision system based on the determined curvature of the road, determines a second relevancy score and a second confidence value associated with the second relevancy score; wherein the vehicular vision system, based on the first relevancy score and the second relevancy score, determines an aggregated relevancy score by weighting the first relevancy score and the second relevancy score, wherein a higher weight is applied to the first relevancy score or the second relevancy score based on the first confidence value and the second confidence value; wherein the vehicular vision system, based at least in part on the aggregated relevancy score, identifies a target traffic light from the plurality of traffic lights that is associated with the traffic lane along which the equipped vehicle is traveling as the vehicle approaches the intersection; wherein the vehicular vision system determines a status of the identified target traffic light via processing at the ECU of image data captured by the camera; and wherein the vehicular vision system, responsive to determining the status of the identified target traffic light, performs at least one selected from the group consisting of (i) alerting a driver of the equipped vehicle of status of the target traffic light and (ii) controlling speed of the equipped vehicle. ### Claim 20 (depends on 19) The vehicular vision system of claim 19, wherein the vehicular vision system determines the first relevancy score based at least in part on (i) a longitudinal distance of the target traffic light from the equipped vehicle and (ii) a lateral distance of the target traffic light from the equipped vehicle. ### Claim 21 (depends on 19) The vehicular vision system of claim 19, wherein the vehicular vision system determines the first relevancy score based at least in part on vehicle dynamics data. ### Claim 22 (depends on 19) The vehicular vision system of claim 19, wherein the vehicular vision system determines the second relevancy score based at least in part on global positioning system data. ### Claim 23 (depends on 19) The vehicular vision system of claim 19, wherein the vehicular vision system determines the second relevancy score based at least in part on a longitudinal distance between the equipped vehicle and the intersection. ### Claim 24 (depends on 19) The vehicular vision system of claim 19, wherein the vehicular vision system determines topology information for the traffic lane based on processing at the ECU of image data captured by the camera, and wherein the first relevancy score is further based on the topology information. ### Claim 25 (depends on 19) The vehicular vision system of claim 19, wherein the vehicular vision system determines the aggregated relevancy score by weighting the first relevancy score and the second relevancy score. ### Claim 26 (depends on 19) The vehicular vision system of claim 19, wherein the vehicular vision system, based on the aggregated relevancy score and vehicle dynamics data, determines at least one selected from the group consisting of (i) a stopping distance for the equipped vehicle that will stop the vehicle prior to entering the intersection and (ii) an amount of time remaining until the equipped vehicle will enter the intersection. ### Claim 27 A vehicular vision system, the vehicular vision system comprising: a camera disposed at a windshield of a vehicle equipped with the vehicular vision system, the camera viewing at least forward of the equipped vehicle through the windshield of the equipped vehicle; wherein the camera is operable to capture image data; wherein the camera comprises a CMOS imaging array, and wherein the CMOS imaging array comprises at least one million photosensors arranged in rows and columns; an electronic control unit (ECU) comprising electronic circuitry and associated software; wherein image data captured by the camera is transferred to the ECU; wherein the electronic circuitry of the ECU comprises at least one image processor; wherein the ECU is operable to process image data captured by the camera and transferred to the ECU; wherein, as the vehicle travels along a traffic lane of a road, the vehicular vision system, (i) via processing at the ECU of image data captured by the camera and (ii) based on map data, determines presence of a plurality of traffic lights at an intersection that the equipped vehicle is approaching, and wherein each traffic light of the plurality of traffic lights is associated with one or more traffic lanes of the road at the intersection; wherein the vehicular vision system, based on processing at the ECU of image data captured by the camera, determines a first relevancy score and a first confidence value associated with the first relevancy score; wherein the vehicular vision system, based on the map data, determines a curvature of the road; wherein the vehicular vision system, based on the determined curvature of the road, determines a second relevancy score and a second confidence value associated with the second relevancy score; wherein the vehicular vision system, based on the first relevancy score and the second relevancy score, determines an aggregated relevancy score by weighting the first relevancy score and the second relevancy score, wherein a higher weight is applied to the first relevancy score or the second relevancy score based on the first confidence value and the second confidence value; wherein the vehicular vision system, based at least in part on the aggregated relevancy score, identifies a target traffic light from the plurality of traffic lights that is associated with the traffic lane along which the equipped vehicle is traveling as the vehicle approaches the intersection; wherein the vehicular vision system determines a color of an illuminated indicator of the identified target traffic light via processing at the ECU of image data captured by the camera; and wherein the vehicular vision system, responsive to determining the color of the illuminated indicator, alerts a driver of the equipped vehicle of the color of the illuminated indicator. ### Claim 28 (depends on 27) The vehicular vision system of claim 27, wherein the alert comprises at least one selected from the group consisting of (i) a visual notification, (ii) an audible notification and (iii) a haptic notification. ### Claim 29 (depends on 28) The vehicular vision system of claim 28, wherein the alert comprises the visual notification, and wherein the visual notification includes the color of the illuminated indicator of the identified target traffic light. ### Claim 30 (depends on 27) The vehicular vision system of claim 27, wherein the vehicular vision system determines a change in the color of the illuminated indicator associated with the identified target traffic light, and wherein the vehicular vision system, responsive to the determined change in the color of the illuminated indicator, generates a second alert for the driver of the equipped vehicle.