Method for distress and road rage detection

A system for determining distress of a driver of a vehicle, the system comprising:

• a plurality of sensors, including interior vehicle image sensors, an interior vehicle audio sensor, vehicle data sensors, and Global Positioning System (GPS) data sensors;
• and
• one or more processors in communication with the plurality of sensors, the one or more processors configured to:
• receive inputs from the plurality of sensors;
• process the received inputs to obtain a driver heat change estimate, a driver expression estimate, a driver gesture estimate, an on-board diagnostics (OBD) estimate, and a GPS estimate;
• store the estimates in a memory;
• use the stored estimates to generate deviation scores for each of the driver heat change estimate, the driver expression estimate, the driver gesture estimate, the OBD estimate, and the GPS estimate, wherein generating the deviation score for the driver expression estimate includes:
• using detection-tracking-validation (DTV) to localize frontal face images of the driver;
• constructing a face stream frame from a partitioned face region of the frontal face images;
• applying a fully convolutional network (FCN) to the face stream frame using an encoder, including using multiple convolutional, pooling, batch normalization, and rectified linear unit (ReLU) layers;
• reshaping a feature map of a last layer of the encoder into vector form to obtain an output, and applying the output to a recurrent neural network (RNN) to obtain a normal driving expression model using a Gaussian mixture model (GMM);
• and
• comparing a real-time driver or passenger expression with the normal driving expression model to calculate the deviation score for the driver expression estimate;
• execute a machine learning algorithm to classify driver behavior as normal or impaired based on the deviation scores;
• and
• generate a warning based on the classification indicating impaired driver behavior.

A non-transitory computer-readable medium storing computer instructions to determine distress of a driver of a vehicle and provide a warning, that when executed by one or more processors, cause the one or more processors to perform steps of:

• receiving inputs from a plurality of sensors, including interior vehicle image sensors, an interior vehicle audio sensor, vehicle data sensors, and Global Positioning System (GPS) data sensors;
• processing the received inputs to obtain a driver heat change estimate, a driver expression estimate, a driver gesture estimate, an on-board diagnostics (OBD) estimate, and a GPS estimate;
• storing the estimates in a memory;
• using the stored estimates to generate deviation scores for each of the driver heat change estimate, the driver expression estimate, the driver gesture estimate, the OBD estimate, and the GPS estimate, wherein generating the deviation score for the driver gesture estimate includes:
• detecting driver gestures to obtain an image of a hands region of the driver;
• constructing a two-layer hand stream from the image and normalizing the two-layer hand stream for size adjustment;
• applying a fully convolutional network (FCN) to the two-layer hand stream using an encoder, including using multiple convolutional, pooling, batch normalization, and rectified linear unit (ReLU) layers;
• reshaping a feature map of a last layer of the encoder into vector form to obtain an output, and applying the output to a recurrent neural network (RNN) to obtain a normal driving gesture model using a Gaussian mixture model (GMM);
• and
• comparing a real-time driver gesture with the normal driving gesture model to calculate the deviation score for the driver gesture estimate;
• executing a machine learning algorithm to classify driver behavior as normal or impaired based on the deviation scores;
• and
• generating the warning based on the classification indicating impaired driver behavior.

A method for to determine distress of a driver of a vehicle, the method comprising:

• receiving inputs by one or more processors from a plurality of sensors, including interior vehicle image sensors, an interior vehicle audio sensor, vehicle data sensors, and Global Positioning System (GPS) data sensors;
• processing the received inputs by the one or more processors to obtain a driver heat change estimate, a driver expression estimate, a driver gesture estimate, an on-board diagnostics (OBD) estimate, and a GPS estimate;
• storing by the one or more processors the estimates in a memory;
• using the stored estimates by the one or more processors to generate deviation scores for each of the driver heat change estimate, the driver expression estimate, the driver gesture estimate, the OBD estimate, and the GPS estimate;
• executing a machine learning algorithm by the one or more processors to classify driver behavior as normal or impaired based on the deviation scores using a Gaussian mixture model (GMM), wherein expectation maximization is used to estimate model parameters of the GMM;
• and
• generating a warning by the one or more processors based on the classification indicating impaired driver behavior.