# US 2020/0005060 A1 · Machine learning based driver assistance

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0005060 A1
- Document ID: US20200005060A1
- Status: Published application
- Kind: A1
- Application number: 16/481,822
- Filed: 2018-01-31
- Published: 2020-01-02
- Earliest claimed: 2017-01-31
- Assignees: The Regents of the University of California
- Applicants: The Regents of the University of California
- Inventors: Sujitha Martin; Kevan Yuen; Mohan M. Trivedi
- Web page: https://patentgrep.com/patent/US20200005060A1

## CPC

- G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness}
- B60W 40/08: related to drivers or passengers
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V 40/197: {Matching; Classification}
- B60W 2040/0827: {due to sleepiness}
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2540/223: Posture, e.g. hand, foot, or seat position, turned or inclined
- B60W 2540/225: Direction of gaze
- B60W 2540/229: Attention level, e.g. attentive to driving, reading or sleeping
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method for machine learning based driver assistance is provided. The method may include detecting, in one or more images of a driver operating an automobile, one or more facial landmarks. The detection of the one or more facial landmarks may include applying, to the one or more images, a first machine learning model. A gaze dynamics of the driver may be determined based at least on the one or more facial landmarks. The gaze dynamics of the driver may include a change in a gaze zone of the driver from a first gaze zone to a second gaze zone. A state of the driver may be determined based at least on the gaze dynamics of the driver. An operation of the automobile may be controlled based at least on the state of the driver. Related systems and articles of manufacture, including computer program products, are also provided.

## Claims

### Claim 1

A system, comprising:
at least data processor; and
at least one memory storing instructions which, when executed by the at least one data processor, result in operations comprising:
detecting, in one or more images of a driver operating an automobile, one or more facial landmarks, the detection of the one or more facial landmarks comprising applying, to the one or more images, a first machine learning model;
determining, based at least on the one or more facial landmarks, a gaze dynamics of the driver, the gaze dynamics of the driver comprising a change in a gaze zone of the driver from a first gaze zone to a second gaze zone;
determining, based at least on the gaze dynamics of the driver, a state of the driver; and
controlling, based at least on the state of the driver, an operation of the automobile.

### Claim 2 (depends on 1)

The system of claim 1, wherein the one or more facial landmarks includes a pupil, an iris, and/or an eyelid.

### Claim 3 (depends on 1)

The system of claim 1, further comprising:
training, based at least on training data, the first machine learning model, the training data including a plurality of images that includes and/or excludes the one or more facial landmarks, and the first machine learning model being trained to detect the one or more facial landmarks in the plurality of images.

### Claim 4 (depends on 1)

The system of claim 1, further comprising:
detecting, in the one or more images of the driver operating the automobile, a face, the detection of the face comprising applying, to the one or more images, a second machine learning model.

### Claim 5 (depends on 4)

The system of claim 4, further comprising:
training, based at least on training data, the second machine learning model, the training data including a plurality of images that includes and/or excludes the face, and the second machine learning model being trained to detect the face in the plurality of images.

### Claim 6 (depends on 1)

The system of claim 1, wherein the first gaze zone and/or the second gaze zone comprises different areas at which the driver looks while operating the automobile.

### Claim 7 (depends on 1)

The system of claim 1, wherein the first gaze zone and/or the second gaze zone corresponds to a left side mirror of the automobile, a right side mirror of the automobile, a rear view mirror of the automobile, an instrument cluster of the automobile, a center console of the automobile, a first portion of a windshield directly in front of the driver, a second portion of the windshield to a right of the driver, and/or a driver side window.

### Claim 8 (depends on 1)

The system of claim 1, wherein the determination of the state of the driver comprises comparing the gaze dynamics of the driver to at least one gaze model associated with a maneuver, and wherein the at least one gaze model comprises gaze zones observed in drivers performing the maneuver.

### Claim 9 (depends on 8)

The system of claim 8, wherein the at least one gaze model comprises a Gaussian distribution of the gaze zones observed in drivers performing the maneuver.

### Claim 10 (depends on 8)

The system of claim 8, wherein the at least one gaze model comprises gaze zones observed in drivers before an execution of the maneuver, during the execution of the maneuver, and/or after the execution of the maneuver.

### Claim 11 (depends on 1)

The system of claim 1, wherein the first gaze zone and/or the second gaze zone corresponds to one or more eyes of the driver being closed and/or cast downward, and wherein the state of the driver comprises that the driver is inattentive.

### Claim 12 (depends on 1)

The system of claim 1, wherein the first machine learning model comprises a neural network, a regression model, an instance-based model, a regularization model, a decision tree, a Bayesian model, a clustering model, an associative model, a deep learning model, a dimensionality reduction model, and/or an ensemble model.

### Claim 13 (depends on 1)

The system of claim 1, wherein the first machine learning model is configured to output a heat map comprising a first location having a first color and a second location having a second color, wherein the first location and the second location comprises different portions of the one or more images, wherein the first color corresponds to a first probability of the one or more facial landmarks being present at the first location, and wherein the second color corresponds to a second probability of the one or more facial landmarks being present at the second location.

### Claim 14 (depends on 1)

The system of claim 1, wherein the control of the operation of the automobile comprises steering, acceleration, deceleration, and/or braking.

### Claim 15 (depends on 1)

The system of claim 1, further comprising:
determining, based at least on the one or more images of the driver, a head pose of the driver, the head pose of the driver comprising a position and/or an orientation of a head of the driver, and the gaze dynamics of the driver further being determined based at least on the head pose of the driver.

### Claim 16

A computer-implemented method, comprising:
detecting, in one or more images of a driver operating an automobile, one or more facial landmarks, the detection of the one or more facial landmarks comprising applying, to the one or more images, a first machine learning model;
determining, based at least on the one or more facial landmarks, a gaze dynamics of the driver, the gaze dynamics of the driver comprising a change in a gaze zone of the driver from a first gaze zone to a second gaze zone;
determining, based at least on the gaze dynamics of the driver, a state of the driver; and
controlling, based at least on the state of the driver, an operation of the automobile.

### Claim 17 (depends on 16)

The method of claim 16, wherein the one or more facial landmarks includes a pupil, an iris, and/or an eyelid.

### Claim 18 (depends on 16)

The method of claim 16, further comprising:
training, based at least on training data, the first machine learning model, the training data including a plurality of images that includes and/or excludes the one or more facial landmarks, and the first machine learning model being trained to detect the one or more facial landmarks in the plurality of images.

### Claim 19 (depends on 16)

The method of claim 16, further comprising:
detecting, in the one or more images of the driver operating the automobile, a face, the detection of the face comprising applying, to the one or more images, a second machine learning model.

### Claim 20 (depends on 19)

The method of claim 19, further comprising:
training, based at least on training data, the second machine learning model, the training data including a plurality of images that includes and/or excludes the face, and the second machine learning model being trained to detect the face in the plurality of images.

### Claim at position 21

21-33. (canceled)
