US 11,386,325 B1Granted

Ensemble neural network state machine for detecting distractions

AssigneeSamsara Inc.

InventorsSharan Srinivasan, Brian Tuan, John Bicket, Jing Wang, Muhammad Ali Akhtar, Abner Ayala Acevedo, Bruce Kellerman, Vincent Shieh

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

A vehicle device may execute one or more neural networks (and/or other artificial intelligence), based on input from one or more of the cameras and/or other sensors, to intelligently detect safety events in real-time. The one or more neural networks may be an ensemble neural network that includes neural networks for detecting a head and hand of a user, neural networks for detecting hand actions of the user, neural networks for detecting the head pose of the user, neural networks for predicting an occurrence of an event, and neural networks for predicting a start time and end time of the event. Further, the neural networks can be segmented into a modular neural network based on metadata. The segmentation of the neural network can define a thin layer of the modular neural network to enable independent tuning of the thin layer of the modular neural network.

Claims20

Dependent claims

A vehicle device comprising:

a computer readable storage medium having program instructions embodied therewith;
and
one or more processors configured to execute the program instructions to cause the vehicle device to:
access sensor data from one or more sensors associated with a vehicle, the sensor data associated with an image of a scene;
execute an ensemble neural network configured to detect occurrence of a safety event associated with a driver of the vehicle, wherein the safety event indicates a distracted state of the driver of the vehicle, the ensemble neural network comprising a plurality of models including:
a first model configured to detect a face and a hand of a user of the vehicle based at least in part on the sensor data,
a second model configured to detect one or more hand actions of the user based at least in part on the detected hand of the user,
a third model configured to detect a head pose of the user based at least in part on the detected face of the user,
a fourth model configured to detect a gaze of the user based at least in part on the detected face of the user, and
a fifth model configured to predict, based at least in part on the one or more hand actions, the head pose, and the gaze, a probability of a safety event associated with the user of the vehicle;
and
in response to determining at least that the probability of the safety event exceeds a threshold, trigger an event alert indicative of occurrence of the safety event.

9 more elements

A vehicle device comprising:

a computer readable storage medium having program instructions embodied therewith;
and
one or more processors configured to execute the program instructions to cause the vehicle device to:
access sensor data from one or more sensors associated with a vehicle;
execute an ensemble neural network configured to detect occurrence of a safety event associated with a driver of the vehicle, the ensemble neural network comprising a plurality of models including:
a first model configured to detect a face and a hand of a user of the vehicle based at least in part on the sensor data,
a second model configured to detect one or more hand actions of the user based at least in part on the detected hand of the user,
a third model configured to detect a head pose of the user based at least in part on the detected face of the user, and
a fourth model configured to predict, based at least in part on the one or more hand actions and the head pose, a probability of the safety event associated with the user of the vehicle;
and
in response to determining at least that the probability of the safety event exceeds a threshold, trigger an event alert indicative of occurrence of the safety event.

8 more elements

Dependencies are inferred from the claim text.