# US 11,993,277 B2 · Inward/outward vehicle monitoring for remote reporting and in-cab warning enhancements > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,993,277 B2 - Document ID: US11993277B2 - Status: Granted - Kind: B2 - Application number: 18/124,351 - Filed: 2023-03-21 - Published: 2023-07-20 - Granted: 2024-05-28 - Earliest claimed: 2018-09-11 - Assignees: Netradyne, Inc. - Applicants: Netradyne, Inc. - Inventors: David Jonathan Julian; Venkata Sreekanta Reddy Annapureddy; Sandeep Pandya; Arvind Yedla; Dale Alan Willis; Venkata Ramanan Venkatachalam Jayaraman; Suresh Babu Yanamala; Avneesh Agrawal; Veeresh Taranalli; Jaeyoon Kim - Web page: https://patentgrep.com/patent/US11993277B2 ## CPC - B60W 50/14: Means for informing the driver, warning the driver or prompting a driver intervention - B60W 40/02: related to ambient conditions - B60W 40/08: related to drivers or passengers - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness} - G06V 40/165: {using facial parts and geometric relationships} - G06V 40/19: Sensors therefor - G08B 21/06: indicating a condition of sleep, e.g. anti-dozing alarms - G08G 1/164: {Centralised systems, e.g. external to vehicles} - B60W 2040/0827: {due to sleepiness} - B60W 2050/143: {Alarm means} - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2520/00: Input parameters relating to overall vehicle dynamics - B60W 2540/225: Direction of gaze - B60W 2540/229: Attention level, e.g. attentive to driving, reading or sleeping - B60W 2552/53: Road markings, e.g. lane marker or crosswalk - B60W 2554/404: Characteristics - B60W 2555/60: Traffic rules, e.g. speed limits or right of way - G06V 40/168: {Feature extraction; Face representation} - G06V 40/18: Eye characteristics, e.g. of the iris - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS - G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93}) ## Abstract Systems and methods are provided for intelligent driving monitoring systems, advanced driver assistance systems and autonomous driving systems, and providing alerts to the driver of a vehicle, based on anomalies detected between driver behavior and environment captured by the outward facing camera. Various aspects of the driver, which may include his direction of sight, point of focus, posture, gaze, is determined by image processing of the upper visible body of the driver, by a driver facing camera in the vehicle. Other aspects of environment around the vehicle captured by the multitude of cameras in the vehicle are used to correlate driver behavior and actions with what is happening outside to detect and warn on anomalies, prevent accidents, provide feedback to the driver, and in general provide a safer driver experience. ## Claims ### Claim 1 A method comprising: capturing, by at least one processor of a computing device with an outward facing camera on or in a vehicle, visual data outside of the vehicle; detecting, by the at least one processor based on the visual data, a red light turned to a green light for the vehicle; responsive to detecting, by the at least one processor, that the vehicle has not moved for at least a threshold period of time since detecting the red light turned to the green light, generating, by the at least one processor, an alert for a driver of the vehicle; and responsive to detecting, by the at least one processor, a sound of a horn after detecting the red light turned to the green light, extending, by the at least one processor, the threshold period of time. ### Claim 2 (depends on 1) The method of claim 1, further comprising responsive to detecting, by the at least one processor, a sound of a siren after detecting the red light turned to the green light, suppressing, by the at least one processor, the alert to the driver of the vehicle. ### Claim 3 A method comprising: capturing, by at least one processor of a computing device with an outward facing camera on or in a vehicle, visual data outside of the vehicle; detecting, by the at least one processor based on the visual data, a red light turned to a green light for the vehicle, wherein detecting that a driver of the vehicle is not looking at the green light comprises detecting, using a microphone, sound inside of the vehicle; and responsive to detecting, by the at least one processor, that the driver of the vehicle is not looking at the green light, generating, by the at least one processor, an alert for a driver of the vehicle. ### Claim 4 (depends on 3) The method of claim 3, wherein detecting that the driver of the vehicle is not looking at the green light comprises detecting that the driver is looking into the cab of the vehicle, shoulder checking, eating, interacting with a dashboard of the vehicle, using a phone, gazing at a phone, texting, talking to a passenger of the vehicle, sleeping, appearing drowsy, or handling children. ### Claim 5 (depends on 3) The method of claim 3, wherein the microphone is inside of the vehicle. ### Claim 6 (depends on 3) The method of claim 3, wherein the microphone is outside of the vehicle. ### Claim 7 (depends on 6) The method of claim 6, wherein sound outside of the vehicle comprises a siren, a car screeching, or a horn honking. ### Claim 8 (depends on 7) The method of claim 7, further comprising responsive to detecting, by the at least one processor, the horn honking after detecting the red light turned to the green light, transmitting, by the at least one processor, the alert after a period of delay. ### Claim 9 (depends on 3) The method of claim 3, further comprising responsive to detecting, by the at least one processor, the siren after detecting the red light turned to the green light, suppressing, by the at least one processor, the alert to the driver of the vehicle. ### Claim 10 A method comprising: capturing, by at least one processor of a computing device with an outward facing camera on or in a vehicle, visual data outside of the vehicle; detecting, by the at least one processor based on the visual data, a traffic light corresponding to a lane of the vehicle; detecting, by the at least one processor based on the visual data, a red light turned to a green light for the traffic light corresponding to the lane of the vehicle; generating, by the at least one processor, an alert for a driver of the vehicle when the traffic light corresponding to the lane of the vehicle turned to the green light; detecting, by the at least one processor based on the visual data, a second traffic light corresponding to an adjacent lane of the vehicle; detecting, by the at least one processor based on the visual data, a second red light turned to a second green light for the second traffic light corresponding to the adjacent lane; and suppressing, by the at least one processor, the alert for the driver of the vehicle when the second traffic light corresponding to the adjacent lane of the vehicle turned to the green light. ### Claim 11 (depends on 10) The method of claim 10, further comprising responsive to detecting, by the at least one processor, that the vehicle has not moved for at least a threshold period of time since detecting the red light turned to the green light, transmitting, by the at least one processor, an alert to a driver of the vehicle. ### Claim 12 (depends on 11) The method of claim 11, further comprising responsive to detecting, by the at least one processor, a sound of a horn after detecting the red light turned to the green light, extending, by the at least one processor, the threshold period of time. ### Claim 13 A method comprising: capturing, by at least one processor of a computing device with an outward facing camera on or in a vehicle, visual data outside of the vehicle; detecting, by the at least one processor based on the visual data, a traffic light corresponding to a lane of the vehicle; detecting, by the at least one processor based on the visual data, a red light turned to a green light for the traffic light corresponding to the lane of the vehicle; generating, by the at least one processor, an alert for a driver of the vehicle when the traffic light corresponding to the lane of the vehicle turned to the green light; and responsive to detecting, by the at least one processor, a sound of a siren after detecting the red light turned to the green light, suppressing, by the at least one processor, the alert to the driver of the vehicle. ### Claim 14 (depends on 10) The method of claim 10, further comprising detecting, by the at least one processor, that the driver of the vehicle is not looking at the green light. ### Claim 15 (depends on 14) The method of claim 14, wherein detecting that the driver of the vehicle is not looking at the green light comprises detecting that the driver is looking into the cab of the vehicle, shoulder checking, eating, interacting with a dashboard of the vehicle, using a phone, gazing at a phone, texting, talking to a passenger of the vehicle, sleeping, appearing drowsy, or handling children. ### Claim 16 (depends on 14) The method of claim 14, wherein detecting that the driver of the vehicle is not looking at the green light comprises detecting, using a microphone inside of the vehicle, sound inside of the vehicle. ### Claim 17 (depends on 14) The method of claim 14, wherein detecting that the driver of the vehicle is not looking at the green light comprises detecting, using a microphone outside of the vehicle, sound outside of the vehicle. ### Claim 18 (depends on 17) The method of claim 17, wherein sound outside of the vehicle comprises a siren, a car screeching, or a horn honking.