# US 2021/0241011 A1 · Drowsiness Detection System

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241011 A1
- Document ID: US20210241011A1
- Status: Published application
- Kind: A1
- Application number: 17/055,991
- Filed: 2019-10-22
- Published: 2021-08-05
- Earliest claimed: 2018-10-22
- Assignees: 5DT, Inc.
- Applicants: 5DT, Inc.
- Inventors: Jaco Cronje; Johann Ephraim Hough
- Web page: https://patentgrep.com/patent/US20210241011A1

## CPC

- G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness}
- A61B 5/1103: {Detecting muscular movement of the eye, e.g. eyelid movement}
- A61B 5/1114: {Tracking parts of the body}
- A61B 5/1123: {Discriminating type of movement, e.g. walking or running (A61B5/1116, A61B5/112 take precedence)}
- A61B 5/1128: {using image analysis (A61B5/1127 takes precedence)}
- A61B 5/163: {by tracking eye movement, gaze, or pupil change}
- A61B 5/165: {Evaluating the state of mind, e.g. depression, anxiety}
- A61B 5/18: for vehicle drivers {or machine operators}
- A61B 5/6893: {Cars}
- A61B 5/7267: {involving training the classification device}
- B60Q 9/00: Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 {, e.g. haptic signalling}
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G08B 21/06: indicating a condition of sleep, e.g. anti-dozing alarms
- G08B 29/186: {Fuzzy logic; neural networks}
- G08B 29/188: {Data fusion; cooperative systems, e.g. voting among different detectors}
- A61B 2560/0247: for compensation or correction of the measured physiological value
- G06T 2207/30201: Face
- G06T 2207/30268: Vehicle interior
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- B60Q: ARRANGEMENT OF SIGNALLING OR LIGHTING DEVICES, THE MOUNTING OR SUPPORTING THEREOF OR CIRCUITS THEREFOR, FOR VEHICLES IN GENERAL
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS

## Abstract

A machine-implemented method for automated detection of drowsiness, which includes receiving from an imaging device directed at the face of an operator a series of images of the face of the operator onto processing hardware, on the processor detecting facial landmarks of the operator from the series of images to determine the level of talking by the operator, the level of yawning of the operator, the PERCLOS of the operator, on the processor detecting the facial pose of the operator from the series of images to determine the level of gaze fixation by the operator, on the processor calculating the level of drowsiness of the operator by ensembling the level of talking by the operator, the level of yawning of the operator, the PERCLOS of the operator and the level of gaze fixation by the operator, and generating an alarm when the calculated level of drowsiness of the operator exceeds a predefined value.

## Claims

### Claim 1

A drowsiness detection system, which includes:
an image sensor, directed towards a face of an operator for capturing a series of images of a face of the operator;
at least one movement sensor installed in a vehicle to monitor movement of the vehicle;
processing hardware which comprises a central processing unit, a data storage facility in communication with the central processing unit and input/output interfaces in communication with the central processing unit, the processing hardware being configured to implement a set of trained convolution neural networks (CNNs) comprising:
a face detection group into which at least one image of the series of images is received from the image sensor for detecting the face of the operator in the at least one image;
a face tracking group which match the face in the at least one image in the series of images with the same face in a previous image in the series of images and which then track the motion of the same face in the further series of images;
a facial landmark detection group which detects movement of a plurality of facial landmarks in the further series of images, the facial landmarks selected from the group of jawline, eyebrows, eyelids, nose, pupils and lips;
a face pose detection group which determine the orientation of the face of the operator; and
a drowsiness detection group which includes any one or more of:
a yawning and talking detection group, which is trained to determine whether an operator is talking, thereby to provide an indication of alertness, and whether an operator is yawning, thereby to provide an indication of drowsiness;
a PERCLOS (percentage of eye closure over time) calculation, which determines the percentage of eye closure over a period of time, thereby to provide an indication of drowsiness if the PERCLOS exceeds a predefined value;
a fixed gaze calculation, which determines when an operator's gaze time exceeds a predefined value, thereby to provide an indication of drowsiness;
an erratic movement detection group, which is trained to determine when a vehicle movement exceeds a predefined value, thereby to provide an indication of drowsiness.

### Claim 2 (depends on 1)

The drowsiness detection system of claim 1, in which the drowsiness detection group includes an ensemble algorithm, which ensembles outputs from any one or more of the yawning and talking detection group, the PERCLOS calculation, the fixed gaze calculation, and the erratic movement detection group thereby to determine the drowsiness of an operator.

### Claim 3 (depends on 2)

The drowsiness detection system of claim 2, in which the ensemble algorithm calculates the level of drowsiness of an operator by any one of:
predefined algorithms that calculate a numerical value representative of the drowsiness of the operator by using mathematical equations and conditional statements applied to the outputs from the yawning and talking detection group, the PERCLOS detection group, the fixed gaze detection group and the erratic movement detection group; and
a machine learning algorithm trained with outputs from the yawning and talking detection group, the PERCLOS detection group, the fixed gaze detection group and the erratic movement detection group to determine a level of drowsiness of an operator.

### Claim 4 (depends on 3)

The drowsiness detection system of claim 3, in which the ensemble algorithm includes machine learning algorithms selected from any one or more of neural networks, regression tree ensembles, linear regression and polynomial regression.

### Claim 5 (depends on 1)

The drowsiness detection system of claim 1, which includes an ambient light sensor, installed in proximity to the operator, in use to measure ambient light in the environment of the operator to aid the machine learning and computer vision algorithms.

### Claim 6

A machine-implemented method for automated detection of drowsiness, which includes:
receiving onto processing hardware a series of images of an operator,
processing the series of images of the operator on processing hardware by a set of trained convolution neural networks comprising:
a face detection group into which at least one image of the series of images is received from the image sensor for detecting the face of the operator in the at least one image;
a face tracking group which match the face in the at least one image in the series of images with the same face in a previous image in the series of images and which then track the motion of the same face in the further series of images;
a facial landmark detection group which detects movement of a plurality of facial landmarks in the further series of images, the facial landmarks selected from the group of jawline, eyebrows, eyelids, nose, pupils and lips;
a face pose detection group which determine the orientation of the face of the operator; and
a drowsiness detection group, which includes any one or more of:
a yawning and talking detection group, which is trained to determine whether an operator is talking, thereby to provide an indication of alertness, and whether an operator is yawning, thereby to provide an indication of drowsiness;
a PERCLOS (percentage of eye closure over time) calculation, which determines the percentage of eye closure over a period of time, thereby to provide an indication of drowsiness if the PERCLOS exceeds a predefined value;
a fixed gaze calculation, which determines when an operator's gaze time exceeds a predefined value, thereby to provide an indication of drowsiness;
an erratic movement detection group, which is trained to determine when a vehicle's movement resembles that of a drowsy driver, thereby to provide an indication of drowsiness;
ensembling the outputs received from any one or more of the yawning and talking detection group, the PERCLOS calculation, the a fixed gaze calculation, and the erratic movement detection group thereby to generate an alarm when the ensembled value exceed a predefined value for a drowsy operator; and
generating an alarm when the level of drowsiness of the operator determined by the processing hardware exceeds a predefined value.

### Claim 7

A machine-implemented method for automated detection of drowsiness, which includes
receiving from an imaging device directed at the face of an operator a series of images of the face of the operator onto processing hardware,
on the processor detecting facial landmarks of the operator from the series of images to determine the level of talking by the operator, the level of yawning of the operator, the PERCLOS of the operator;
on the processor detecting the facial pose of the operator from the series of images to determine the level of gaze fixation by the operator;
on the processor calculating the level of drowsiness of the operator by ensembling the level of talking by the operator, the level of yawning of the operator, the PERCLOS of the operator and the level of gaze fixation by the operator; and
generating an alarm when the calculated level of drowsiness of the operator exceeds a predefined value.

### Claim 8 (depends on 7)

The machine-implemented method for automated detection of drowsiness as claimed in claim 7, in which the step of detecting facial landmarks of the face of the operator includes the prior steps of:
detecting a face of the operator in the series of images; and
tracking the face of the operator in the series of images.

### Claim 9 (depends on 8)

The machine-implemented method for automated detection of drowsiness as claimed in claim 8, in which the step of detecting facial landmarks of the face of the operator includes detecting any one or more of:
movement of the jawline of the operator in the series of images;
movement of the eyebrows of the operator in the series of images;
movement of the eyelids of the operator in the series of images;
movement of the nose of the operator in the series of images;
movement of the pupils of the operator in the series of images; and
movement of the lips of the operator in the series of images.

### Claim 10 (depends on 7)

The machine-implemented method for automated detection of drowsiness as claimed in claim 7, in which the level of gaze fixation of an operator is detected by calculating the variance of the operator's head orientation and the operator's pupil movement, the variance of both the head's orientation and pupil movement being used to predict the drowsiness level of the operator.