# US 11,027,750 B2 · Method and system for assisting drivers to drive with precaution > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,027,750 B2 - Document ID: US11027750B2 - Status: Granted - Kind: B2 - Application number: 16/426,090 - Filed: 2019-05-30 - Published: 2019-12-05 - Granted: 2021-06-08 - Earliest claimed: 2018-06-01 - Assignees: Volvo Car Corporation - Applicants: Volvo Car Corporation - Inventors: Trent Victor - Web page: https://patentgrep.com/patent/US11027750B2 ## CPC - B60W 50/16: Tactile feedback to the driver, e.g. vibration or force feedback to the driver on the steering wheel or the accelerator pedal - G08G 1/166: {for active traffic, e.g. moving vehicles, pedestrians, bikes} - B60Q 9/008: {for anti-collision purposes} - B60W 30/09: Taking automatic action to avoid collision, e.g. braking and steering - B60W 30/0956: {the prediction being responsive to traffic or environmental parameters} - B60W 40/08: related to drivers or passengers - B60W 50/14: Means for informing the driver, warning the driver or prompting a driver intervention - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness} - G08G 1/165: {for passive traffic, e.g. including static obstacles, trees} - B60W 2040/0818: {Inactivity or incapacity of driver} - B60W 2540/221: Physiology, e.g. weight, heartbeat, health or special needs - B60W 2540/229: Attention level, e.g. attentive to driving, reading or sleeping - G06V 40/161: {Detection; Localisation; Normalisation} - G06V 40/19: Sensors therefor - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93}) - B60Q: ARRANGEMENT OF SIGNALLING OR LIGHTING DEVICES, THE MOUNTING OR SUPPORTING THEREOF OR CIRCUITS THEREFOR, FOR VEHICLES IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method and system operable for detecting and adequately warning a driver that is determined to be of a unique out-of-the-loop class of drivers that mistakenly believe that a road vehicle operating in a driver-assist or self-driving (i.e., piloted-driving) mode will automatically identify and react to a perceived or sudden threat. A method and system operable for detecting and adequately warning an inattentive, drowsy, and/or out-of-the-loop driver and assessing and confirming the driver's adequate response to the associated warning prompt(s), which may include audio, visual, haptic, and, more preferably, brake pulse or the like warning prompt(s). ## Claims ### Claim 1 A method for assisting a driver of a host vehicle to drive with precaution, coincident with or after determining from one or more host vehicle sensors or cameras arranged to monitor an environment of the host vehicle, other road users, and at least one particular feature associated with a traffic situation of the host vehicle, and applying at least one hypothesis related to at least one hypothetical threat that may arise based on the determined other road users and the at least one particular feature, the method comprising: analyzing an out-of-the-loop condition of the host vehicle driver based on one or more of host vehicle driver eye movement, host vehicle driver activity, host vehicle equipment sensor data, and host vehicle driver periods of attention and inattention over a period of time prior to applying the at least one hypothesis related to the at least one hypothetical threat, wherein analyzing the out-of-the-loop condition of the host vehicle driver comprises identifying one or more of inappropriate off-path (or off-threat) long glance durations, inappropriate on-path (or on-threat) short glance durations, and detection of a series of on-and-off glances over a window of time and comparing observed glance patterns to reference patterns of attentive and skilled driver eye movements to determine appropriateness and inappropriateness of the observed glance patterns; setting or estimating, based in part on the out-of-the-loop condition, a host vehicle driver level of attention threshold required to handle the at least one hypothetical threat and estimating a time until that host vehicle driver level of attention will be required; deriving, from one or more host vehicle driver-monitoring sensors of the host vehicle, a current host vehicle driver level of attention; determining if the set or estimated required host vehicle driver level of attention threshold exceeds the current host vehicle driver level of attention; and if the set or estimated required host vehicle driver level of attention threshold exceeds the current host vehicle driver level of attention, producing at least one of a visual, acoustic, haptic information, and brake impulse prompt to a host vehicle driver environment to promote increased host vehicle driver-attention to the at least one hypothetical threat and/or triggering at least one of automated braking and steering of the host vehicle to promote an increase in safety margin to the at least one hypothetical threat. ### Claim 2 (depends on 1) The method of claim 1, further comprising monitoring a host vehicle driver response to the at least one of a visual, acoustic, haptic information, and brake impulse prompt using one or more of host vehicle driver eye movement, host vehicle driver activity, host vehicle equipment sensor data, and host vehicle driver periods of attention and inattention over a period of time and, if the response falls below a predetermined threshold formulated using probability data, providing at least one of an enhanced visual, acoustic, haptic information, and brake impulse prompt to the host vehicle driver. ### Claim 3 (depends on 1) The method of claim 1, further comprising monitoring a host vehicle driver response to the at least one of a visual, acoustic, haptic information, and brake impulse prompt using one or more of host vehicle driver eye movement, host vehicle driver activity, host vehicle equipment sensor data, and host vehicle driver periods of attention and inattention over a period of time and, if the response falls below a predetermined threshold formulated using probability data, causing the host vehicle to automatically take a responsive action. ### Claim 4 (depends on 1) The method of claim 1, wherein analyzing the out-of-the-loop condition of the host vehicle driver comprises patterning and summarizing the host vehicle driver eye movement with a buffer. ### Claim 5 (depends on 4) The method of claim 4, wherein analyzing the out-of-the-loop condition of the host vehicle driver comprises integrating time looking at the road and time looking away from the road over time, with time looking at the road credited and time looking away from the road debited. ### Claim 6 (depends on 1) The method of claim 1, wherein the reference patterns are determined for different driving contexts. ### Claim 7 (depends on 1) The method of claim 1, wherein the reference patterns are adapted for individual differences. ### Claim 8 A non-transitory computer-readable medium comprising instructions stored in a memory that, when executed by a processor, cause the processor to, after or coincident with determining from one or more host vehicle sensors or cameras arranged to monitor an environment of the host vehicle, other road users, and at least one particular feature associated with a traffic situation of the host vehicle, and applying at least one hypothesis related to at least one hypothetical threat that may arise based on the determined other road users and the at least one particular feature, perform the steps comprising: analyzing an out-of-the-loop condition of the host vehicle driver based on one or more of host vehicle driver eye movement, host vehicle driver activity, host vehicle equipment sensor data, and host vehicle driver periods of attention and inattention over a period of time prior to applying the at least one hypothesis related to the at least one hypothetical threat, wherein analyzing the out-of-the-loop condition of the host vehicle driver comprises identifying one or more of inappropriate off-path (or off-threat) long glance durations, inappropriate on-path (or on-threat) short glance durations, and detection of a series of on-and-off glances over a window of time and comparing observed glance patterns to reference patterns of attentive and skilled driver eye movements to determine appropriateness and inappropriateness of the observed glance patterns; setting or estimating, based in part on the out-of-the-loop condition, a host vehicle driver level of attention threshold required to handle the at least one hypothetical threat and estimating a time until that host vehicle driver level of attention will be required; deriving, from one or more host vehicle driver-monitoring sensors of the host vehicle, a current host vehicle driver level of attention; determining if the set or estimated required host vehicle driver level of attention threshold exceeds the current host vehicle driver level of attention; and if the set or estimated required host vehicle driver level of attention threshold exceeds the current host vehicle driver level of attention, producing at least one of a visual, acoustic, haptic information, and brake impulse prompt to a host vehicle driver environment to promote increased host vehicle driver-attention to the at least one hypothetical threat and/or triggering at least one of automated braking and steering of the host vehicle to promote an increase in safety margin to the at least one hypothetical threat. ### Claim 9 (depends on 8) The non-transitory computer-readable medium of claim 8, the steps further comprising monitoring a host vehicle driver response to the at least one of a visual, acoustic, haptic information, and brake impulse prompt using one or more of host vehicle driver eye movement, host vehicle driver activity, host vehicle equipment sensor data, and host vehicle driver periods of attention and inattention over a period of time and, if the response falls below a predetermined threshold formulated using probability data, providing at least one of an enhanced visual, acoustic, haptic information, and brake impulse prompt to the host vehicle driver. ### Claim 10 (depends on 8) The non-transitory computer readable medium of claim 8, the steps further comprising monitoring a host vehicle driver response to the at least one of a visual, acoustic, haptic information, and brake impulse prompt using one or more of host vehicle driver eye movement, host vehicle driver activity, host vehicle equipment sensor data, and host vehicle driver periods of attention and inattention over a period of time and, if the response falls below a predetermined threshold formulated using probability data, causing the host vehicle to automatically take a responsive action. ### Claim 11 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein analyzing the out-of-the-loop condition of the host vehicle driver comprises patterning and summarizing the host vehicle driver eye movement with a buffer. ### Claim 12 (depends on 11) The non-transitory computer-readable medium of claim 11, wherein analyzing the out-of-the-loop condition of the host vehicle driver comprises integrating time looking at the road and time looking away from the road over time, with time looking at the road credited and time looking away from the road debited. ### Claim 13 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein the reference patterns are determined for different driving contexts. ### Claim 14 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein the reference patterns are adapted for individual differences. ### Claim 15 A vehicle comprising a non-transitory computer-readable medium comprising instructions stored in a memory that, when executed by a processor, cause the processor to, coincident with or after determining from one or more host vehicle sensors or cameras arranged to monitor an environment of the host vehicle, other road users, and at least one particular feature associated with a traffic situation of the host vehicle, and applying at least one hypothesis related to at least one hypothetical threat that may arise based on the determined other road users and the at least one particular feature, perform the steps comprising: analyzing an out-of-the-loop condition of the host vehicle driver based on one or more of host vehicle driver eye movement, host vehicle driver activity, host vehicle equipment sensor data, and host vehicle driver periods of attention and inattention over a period of time prior to applying the at least one hypothesis related to the at least one hypothetical threat, wherein analyzing the out-of-the-loop condition of the host vehicle driver comprises identifying one or more of inappropriate off-path (or off-threat) long glance durations, inappropriate on-path (or on-threat) short glance durations, and detection of a series of on-and-off glances over a window of time and comparing observed glance patterns to reference patterns of attentive and skilled driver eye movements to determine appropriateness and inappropriateness of the observed glance patterns; setting or estimating, based on the applied at least one hypothesis and the out-of-the-loop condition, a host vehicle driver level of attention threshold required to handle the at least one hypothetical threat and estimating a time until that host vehicle driver level of attention will be required; deriving, from one or more host vehicle driver-monitoring sensors of the host vehicle, a current host vehicle driver level of attention; determining if the set or estimated required host vehicle driver level of attention threshold exceeds the current host vehicle driver level of attention; and if the set or estimated required host vehicle driver level of attention threshold exceeds the current host vehicle driver level of attention, producing at least one of a visual, acoustic, haptic information, and brake impulse prompt to a host vehicle driver environment to promote increased host vehicle driver-attention to the at least one hypothetical threat and/or triggering at least one of automated braking and steering of the host vehicle to promote an increase in safety margin to the at least one hypothetical threat. ### Claim 16 (depends on 15) The vehicle of claim 15, wherein the steps performed by the non-transitory computer readable medium further comprise monitoring a host vehicle driver response to the at least one of a visual, acoustic, haptic information, and brake impulse prompt using one or more of host vehicle driver eye movement, host vehicle driver activity, host vehicle equipment sensor data, and host vehicle driver periods of attention and inattention over a period of time and, if the response falls below a predetermined threshold formulated using probability data, providing at least one of an enhanced visual, acoustic, haptic information, and brake impulse prompt to the host vehicle driver. ### Claim 17 (depends on 15) The vehicle of claim 15, wherein the steps performed by the non-transitory computer readable medium further comprise monitoring a host vehicle driver response to the at least one of a visual, acoustic, haptic information, and brake impulse prompt using one or more of host vehicle driver eye movement, host vehicle driver activity, host vehicle equipment sensor data, and host vehicle driver periods of attention and inattention over a period of time and, if the response falls below a predetermined threshold formulated using probability data, causing the host vehicle to automatically take a responsive action. ### Claim 18 (depends on 15) The vehicle of claim 15, wherein analyzing the out-of-the-loop condition of the host vehicle driver comprises patterning and summarizing the host vehicle driver eye movement with a buffer.