System and method extending driver engagement to remote device for autonomous hand-off and eyes-off features

A method to enhance performance and usability of a state machine implementation by an electronic control unit within a vehicle having autonomous operating capabilities, the state machine providing an escalation state used to exit from status quo, the method comprising:

• initiating a driver interactive application to operate during an operation of the vehicle while the vehicle is engaged in at least one of a semi-autonomous or an autonomous vehicle state, wherein the driver interactive application runs in a foreground of an operating system of the hand- held mobile device;
• capturing driver engageability data including image engagement data and eye engagement data tracking from one or more sensors of the hand-held mobile device when the driver interactive application is running in the foreground;
• detecting a facial engagement position of a driver of the vehicle based on the image engagement data;
• detecting an eye engagement position of the driver based on the eye engagement data;
• determining a first driver engageability score based on at least one of the facial engagement position, the eye engagement position, or a combination thereof, wherein the first driver engageability score indicates a level of interaction of the driver with the driver interactive application;
• and
• receiving the first driver engageability score from the hand-held mobile device;
• capturing image tracking data and eye tracking data of the driver while operating the vehicle;
• detecting a tracked facial position of the driver of the vehicle based on the image tracking data;
• detecting a tracked eye position of the driver based on the eye tracking data;
• determining a second driver engageability score based on at least one of the tracked facial position, the tracked eye position, or a combination thereof, wherein the second driver engageability score indicates whether the driver is engaged with the vehicle, alert to environmental conditions of the vehicle, or a combination thereof;
• and
• generating an enhanced engageability score based on the first driver engageability score and the second driver engageability score using an enhanced state machine, wherein the enhanced engageability score augments a decision by the enhanced state machine on whether an escalation state shall occur;
• and wherein the enhanced state machine additionally provides a signal to alert the driver through the hand-held mobile device when the escalation state transition is imminent.

A system to enhance performance and usability of a state machine implementation by an electronic control unit within a vehicle having autonomous operating capabilities, the state machine providing an escalation state used to exit from status quo, the system comprising:

• a hand-held mobile device comprising:
• a first non-transitory memory storing first programming instructions;
• and
• a first processor communicatively coupled to the first non-transitory memory, wherein the first processor executing the first programming instructions is configured to:
• initiate a driver interactive application to operate during an operation of the vehicle while the vehicle is engaged in at least one of a semi-autonomous or an autonomous vehicle state, wherein the driver interactive application runs in a foreground of an operating system of the hand-held mobile device;
• capture multimodal driver engageability data from one or more sensors of the hand-held mobile device when the driver interactive application is initiated, wherein the multimodal driver engageability data includes image engagement data, eye engagement data, and button engagement data;
• detect a facial engagement position of the driver of the vehicle based on the image engagement data;
• detect an eye engagement position of the driver based on the eye engagement data;
• detect usage data associated with one or more hand-held device applications;
• periodically display a digital button on a display screen of the hand-held mobile device;
• detect whether the digital button was pressed to generate button tracking data;
• determine a driver engageability score based on the facial engagement position, the eye engagement position, the usage data, and the button tracking data, wherein the driver engageability score indicates a level of engagement of the driver with the driver interactive application;
• report the driver engageability score to the vehicle;
• and
• a driver monitoring system of the vehicle in communication with the hand-held mobile device, the driver monitoring system comprising:
• a second non-transitory memory storing second programming instructions;
• and
• a second processor communicatively coupled to the second non-transitory memory, wherein the second processor executing the second programming instructions is configured to:
• receive the driver engageability score from the hand-held mobile device;
• capture image tracking data and eye tracking data of the driver while operating the vehicle;
• detect a tracked facial position of the driver of the vehicle based on the image tracking data;
• detect a tracked eye position of the driver based on the eye tracking data;
• determine a driver alertness score based on at least one of the one or more facial engagement positions, the one or more eye engagement positions, or a combination thereof, wherein the driver alertness score indicates whether the driver is engaged with the vehicle, alert to environmental conditions of the vehicle, or a combination thereof;
• and
• generate an enhanced engageability score based on the driver engageability score and the driver alertness score using an enhanced state machine, wherein the multimodal driver engageability data includes tracking data captured of the driver using a background running application on the hand-held mobile device without interaction of the driver, wherein the background running application is configured to monitor the driver while the vehicle operates in an autonomous level of two or higher;
• the enhanced engageability score augments a decision by the enhanced state machine on whether an escalation state transition shall occur;
• and
• the enhanced state machine additionally providing a signal to alert the driver through the hand-held mobile device that the escalation state is imminent.