# US 12,112,555 B1 · Drowsy driving detection > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,555 B1 - Document ID: US12112555B1 - Status: Granted - Kind: B1 - Application number: 18/634,353 - Filed: 2024-04-12 - Granted: 2024-10-08 - Earliest claimed: 2024-04-12 - Assignees: Samsara Inc. - Applicants: Samsara Inc. - Inventors: Sung Chun Lee; Nathan Hurst; Yan Wang; Olamide Akintewe; Justin Levine; Kenshiro Nakagawa; Cole Jurden; Rachel Demerly; Aravindh Ramesh; Kevin Lai; Jovanna Bubar; Shirish Nair; Maisie Wang - Web page: https://patentgrep.com/patent/US12112555B1 ## CPC - G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness} - G06V 10/764: using classification, e.g. of video objects - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/82: using neural networks - G06V 40/161: {Detection; Localisation; Normalisation} - G06V 40/168: {Feature extraction; Face representation} - G06V 40/174: {Facial expression recognition} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Techniques are presented for detecting when drivers drive while drowsy. In some implementations, a drowsiness model is trained with data associated with inward videos and outward videos captured during a trip. The inward videos capture the inside of the cabin with the driver, and the outward videos capture the view in front of the vehicle in the direction of travel. Further, a device at the vehicle periodically calculates a drowsiness scale index value that indicates the level of drowsiness of the driver. Calculating the drowsiness scale index value includes obtaining a set of inward frames from the inward videos; for each inward frame, creating a face image by cropping the inward frame; obtaining a set of outward frames from the outward videos; calculating inward embeddings of the face images and outward embeddings of the outward frames; and calculating, by the drowsiness model, the drowsiness scale index value. ## Claims ### Claim 1 A computer-implemented method comprising: training a drowsiness model with data associated with inward videos and outward videos captured during a trip of a vehicle; and periodically calculating, by a device at the vehicle, a drowsiness scale index value that indicates a level of drowsiness of a driver, wherein calculating the drowsiness scale index value comprises: obtaining a plurality of inward frames from the inward videos; for each inward frame, creating a face image by cropping the inward frame; obtaining a plurality of outward frames from the outward videos; calculating inward embeddings of the face images and outward embeddings of the outward frames; and calculating, by the drowsiness model, the drowsiness scale index value. ### Claim 2 (depends on 1) The method as recited in claim 1, wherein the drowsiness model comprises a face deep feature extraction network and a drowsiness classifier, wherein the face deep feature extraction network calculates the inward embedding for each face image. ### Claim 3 (depends on 2) The method as recited in claim 2, wherein the drowsiness classifier receives as input a plurality of inward embeddings generated by the face deep feature extraction network and a plurality of outward embeddings, wherein the drowsiness classifier outputs the drowsiness scale index value. ### Claim 4 (depends on 1) The method as recited in claim 1, wherein a drowsiness scale comprises values of one for very alert, two for fairly alert, three for signs of sleepiness, four for sleepy, and five for very sleepy. ### Claim 5 (depends on 1) The method as recited in claim 1, wherein the drowsiness model is based on features comprising one or more of inward images from inward videos, outward images from outward videos, trip duration at time of event, inertial measurements, driver identifier, industry identifier, customer identifier, or vehicle type. ### Claim 6 (depends on 1) The method as recited in claim 1, further comprising: detecting a state of drowsiness of the driver based on the drowsiness scale index value; and issuing an audio alert to the driver. ### Claim 7 (depends on 1) The method as recited in claim 1, wherein the inward videos are videos of an inside of a cabin that includes the driver, wherein the outward videos capture a view in front of the vehicle in a direction of travel. ### Claim 8 (depends on 1) The method as recited in claim 1, wherein each calculation of the drowsiness scale index value is based on ten inward embeddings and ten outward embeddings. ### Claim 9 (depends on 1) The method as recited in claim 1, wherein the inward frames are sampled from the inward video at a frequency of two inward frames per second and a window of ten seconds is used to calculate the drowsiness scale index value. ### Claim 10 (depends on 1) The method as recited in claim 1, wherein the drowsiness scale index value is calculated at a predetermined frequency, the predetermined frequency being in a range from once a second to five times a second. ### Claim 11 (depends on 1) The method as recited in claim 1, wherein the drowsiness model includes a ResNet (Residual Network) classifier. ### Claim 12 A system comprising: a memory comprising instructions; and one or more computer processors, wherein the instructions, when executed by the one or more computer processors, cause the system to perform operations comprising: training a drowsiness model with data associated with inward videos and outward videos captured during a trip of a vehicle; and periodically calculating, by a device at the vehicle, a drowsiness scale index value that indicates a level of drowsiness of a driver, wherein calculating the drowsiness scale index value comprises: obtaining a plurality of inward frames from the inward videos; for each inward frame, creating a face image by cropping the inward frame; obtaining a plurality of outward frames from the outward videos; calculating inward embeddings of the face images and outward embeddings of the outward frames; and calculating, by the drowsiness model, the drowsiness scale index value. ### Claim 13 (depends on 12) The system as recited in claim 12, wherein the drowsiness model comprises a face deep feature extraction network and a drowsiness classifier, wherein the face deep feature extraction network calculates the inward embedding for each face image. ### Claim 14 (depends on 13) The system as recited in claim 13, wherein the drowsiness classifier receives as input a plurality of inward embeddings generated by the face deep feature extraction network and a plurality of outward embeddings, wherein the drowsiness classifier outputs the drowsiness scale index value. ### Claim 15 (depends on 12) The system as recited in claim 12, wherein a drowsiness scale comprises values of one for very alert, two for fairly alert, three for signs of sleepiness, four for sleepy, and five for very sleepy. ### Claim 16 (depends on 12) The system as recited in claim 12, wherein the drowsiness model is based on features comprising one or more of inward images from inward videos, outward images from outward videos, trip duration at time of event, inertial measurements, driver identifier, industry identifier, customer identifier, or vehicle type. ### Claim 17 A non-transitory machine-readable storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising: training a drowsiness model with data associated with inward videos and outward videos captured during a trip of a vehicle; and periodically calculating, by a device at the vehicle, a drowsiness scale index value that indicates a level of drowsiness of a driver, wherein calculating the drowsiness scale index value comprises: obtaining a plurality of inward frames from the inward videos; for each inward frame, creating a face image by cropping the inward frame; obtaining a plurality of outward frames from the outward videos; calculating inward embeddings of the face images and outward embeddings of the outward frames; and calculating, by the drowsiness model, the drowsiness scale index value. ### Claim 18 (depends on 17) The non-transitory machine-readable storage medium as recited in claim 17, wherein the drowsiness model comprises a face deep feature extraction network and a drowsiness classifier, wherein the face deep feature extraction network calculates the inward embedding for each face image. ### Claim 19 (depends on 18) The non-transitory machine-readable storage medium as recited in claim 18, wherein the drowsiness classifier receives as input a plurality of inward embeddings generated by the face deep feature extraction network and a plurality of outward embeddings, wherein the drowsiness classifier outputs the drowsiness scale index value. ### Claim 20 (depends on 17) The non-transitory machine-readable storage medium as recited in claim 17, wherein a drowsiness scale comprises values of one for very alert, two for fairly alert, three for signs of sleepiness, four for sleepy, and five for very sleepy.