# US 10,318,831 B2 · Method and system for monitoring the status of the driver of a vehicle > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,318,831 B2 - Document ID: US10318831B2 - Status: Granted - Kind: B2 - Application number: 15/216,177 - Filed: 2016-07-21 - Published: 2018-01-25 - Granted: 2019-06-11 - Earliest claimed: 2016-07-21 - Assignees: Gestigon GmbH - Applicants: gestigon GmbH - Inventors: Sascha Klement; Sebastian Low; Dennis Munster - Web page: https://patentgrep.com/patent/US10318831B2 ## CPC - G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness} - G06F 3/012: {Head tracking input arrangements} - G06F 16/51: Indexing; Data structures therefor; Storage structures - G06F 16/5838: {using colour} - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06V 40/161: {Detection; Localisation; Normalisation} - H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence) - G06T 2207/10012: Stereo images - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A system and method are disclosed relating to a computer system that estimates the status of the driver of a vehicle. The system comprises the following steps: Acquisition of an image from a depth sensor containing depth and optional an IR intensity image and an RGB color image; identification of pixels that belong to the drivers head; creation of a 3D model of the head including an intensity model and a variability estimate for depth, grayscale and color information; estimation of the principal head pose and the neutral facial expression; estimation of the current relative head pose with respect to the principal head pose; identification of pixels that do not match the neutral face model with respect to depth, grayscale or color information or any combination thereof; clustering of the pixels with identified deviations; classification of spatial and temporal patterns to identify driver status and distraction events. ## Claims ### Claim 1 A method for real-time driver state monitoring comprising: operating an image processing system to receive a sequence of depth images from a depth sensor, each depth image comprising image data representing an image of a scene, wherein said image data comprises a depth value indicating distance from the depth sensor; the image processing system identifying a human head in the scene using the sequence of images; the image processing system using the depth images to track position and orientation of the human head with respect to a principal position and orientation where the human head is modeled as a 3D surface to define a head model and continuously adapted over time to obtain an average neutral face model where the neutral face is stored in a database including history of changes; the image processing system identifying depth deviations from the neutral face; and the image processing system classifying such depth deviations as different driver states, where depth variations are compared to the history of changes that has been stored, for a specific driver, in the database and where the comparison is used to extract features for driver-state detection. ### Claim 2 (depends on 1) The method as claimed in claim 1 where supplementary images comprising gray-scale or color images are obtained from the depth sensor or an additional sensor that has been pixel-wise registered to the depth image. ### Claim 3 (depends on 1) The method as claimed in claim 1 where all but head pixels are excluded according to a CAD model of an interior of a car. ### Claim 4 (depends on 1) The method as claimed in claim 1 where the human head is identified by applying a random decision forest. ### Claim 5 (depends on 1) The method as claimed in claim 1 where the human head is identified by skin color. ### Claim 6 (depends on 1) The method as claimed in claim 1 where the human head is identified by a face detection method. ### Claim 7 (depends on 1) The method as claimed in claim 1 where the head model is obtained by an iterative closest point match. ### Claim 8 A system for real-time driver state monitoring, comprising: a depth sensor to capture depth images of a scene and generate image data representing the image of the scene, wherein said image data comprises a depth value indicating distance from the depth sensor; and an image processing system operatively connected to said sensor to receive a sequence of depth images from the sensor, the image processing system being programmed to identify a human head in the scene using the sequence of images, using the depth images to track position and orientation of the human head with respect to a principal position and orientation where the human head is modeled as a 3D surface to define a head model and continuously adapted over time to obtain an average neutral face model, identifying depth deviations from the neutral face, and classifying such depth deviations as different driver states, and wherein the image processing system is programmed whereby the neutral face is stored in a database including history of changes and whereby depth variations are compared to the history of changes that has been stored, for a specific driver, in the database and where the comparison is used to extract features for driver-state detection. ### Claim 9 (depends on 8) The system as claimed in claim 8 where the image processing system receives supplementary images comprising gray-scale or color images from the depth sensor or an additional sensor that has been pixel-wise registered to the depth image. ### Claim 10 (depends on 8) The system as claimed in claim 8 where the image processing system is programmed whereby all but head pixels are excluded according to a CAD model of an interior of a car. ### Claim 11 (depends on 8) The system as claimed in claim 8 where the image processing system is programmed whereby the human head is identified by applying a random decision forest. ### Claim 12 (depends on 8) The system as claimed in claim 8 where the image processing system is programmed whereby the human head is identified by skin color. ### Claim 13 (depends on 8) The system as claimed in claim 8 where the image processing system is programmed whereby the human head is identified by a face detection method. ### Claim 14 (depends on 8) The system as claimed in claim 8 where the image processing system is programmed whereby the head model is obtained by an iterative closest point match.