# US 2020/0394390 A1 · Apparatus and method for vehicle driver recognition and applications of same

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0394390 A1
- Document ID: US20200394390A1
- Status: Published application
- Kind: A1
- Application number: 16/439,870
- Filed: 2019-06-13
- Published: 2020-12-17
- Earliest claimed: 2019-06-13
- Assignees: Xmotors.Ai Inc.
- Applicants: XMotors.ai Inc.
- Inventors: Cong Zhang; Tianpeng Feng; Cheng Lu; Yandong Guo; Jun Ma
- Web page: https://patentgrep.com/patent/US20200394390A1

## CPC

- G06V 20/59: inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
- G06F 18/21: Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V 40/166: {using acquisition arrangements}
- G06V 40/168: {Feature extraction; Face representation}
- G06V 40/172: {Classification, e.g. identification}
- H04N 23/21: from near infrared [NIR] radiation
- H04N 23/56: provided with illuminating means
- H04N 23/74: by influencing the scene brightness using illuminating means
- H04N 23/80: Camera processing pipelines; Components thereof
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Apparatus for vehicle driver recognition includes: a NIR LED illuminator, configured to emit NIR light in the vehicle; a NIR light sensing unit, configured to capture reflected NIR light; an image controlling and processing unit, configured to coordinate the NIR LED illuminator and the NIR light sensing unit, and analyze the reflected NIR light to generate an image; a face detector, configured to determine that a human face exists in the image, and identify a face region; a face feature extractor, configured to analyze the face region to extract a feature vector representing the face region; a face feature dictionary, configured to store existing feature vectors; a face retrieval system, configured to generate an identification result, indicating whether a similarity between the feature vector and any of the existing feature vectors is greater than a first threshold; and a user interface, configured to display the identification result.

## Claims

### Claim 1

An apparatus for vehicle driver recognition, comprising:
a near infrared (NIR) light-emitting diode (LED) illuminator, configured to emit NIR light in the vehicle;
a near infrared (NIR) light sensing unit, configured to capture reflected NIR light;
an image controlling and processing unit, configured to coordinate the NIR LED illuminator and the NIR light sensing unit, and analyze the reflected NIR light captured by the NIR light sensing unit to generate an image;
a face detector, configured to determine that a human face exists in the image, and identify a face region of the human face;
a face feature extractor, configured to analyze the face region to extract a feature vector representing the face region;
a face feature dictionary, configured to store existing feature vectors;
a face retrieval system, configured to generate an identification result, the identification result indicating whether a similarity between the feature vector and any of the existing feature vectors is greater than a first threshold; and
a user interface, configured to display the identification result.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the NIR light sensing unit is a focal plane array (FPA) NIR light sensing unit.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein the NIR light sensing unit is covered with a color filter, the color filter having a passing band between 825 nm and 875 nm.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein the NIR light sensing unit is covered with a color filter, the color filter having a passing band between 915 nm and 965 nm.

### Claim 5 (depends on 1)

The apparatus of claim 1, wherein the image controlling and processing unit is configured to coordinate the NIR LED illuminator and the NIR light sensing unit by controlling one or more of:
a duty cycle of the NIR LED illuminator;
an analog gain of the NIR light sensing unit;
a digital gain of the NIR light sensing unit;
an exposure time of the NIR light sensing unit; and
a frame rate of the NIR light sensing unit.

### Claim 6 (depends on 5)

The apparatus of claim 5, wherein the image controlling and processing unit is configured to coordinate the NIR LED illuminator and the NIR light sensing unit to generate an image with optimal imaging quality.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein the face detector is configured to employ a deep neural network (DNN) to determine that a human face exists in the image, and identify a face region of the human face.

### Claim 8 (depends on 7)

The apparatus of claim 7, wherein the deep neural network is a Multi-Task Convolutional Neural Network (MTCNN).

### Claim 9 (depends on 7)

The apparatus of claim 7, wherein the deep neural network is a Fast Region-based Convolutional Neural Network (Fast R-CNN).

### Claim 10 (depends on 1)

The apparatus of claim 1, further comprising:
a face alignment unit, configured to calibrate the face region to a calibrated face region associated with an upright pose of a driver, wherein the face feature extractor is configured to analyze the calibrated face region to extract a feature vector representing the calibrated face region.

### Claim 11 (depends on 1)

The apparatus of claim 1, wherein the face feature extractor is configured to employ one or more of:
a backbone network;
a local feature descriptor;
a clustering technique; and
a dimension reduction technique.

### Claim 12 (depends on 1)

The apparatus of claim 1, wherein the similarity is a Cosine similarity.

### Claim 13

A method for vehicle driver recognition, comprising:
emitting, by a near infrared (NIR) light-emitting diode (LED) illuminator, NIR light in the vehicle;
capturing, by a near infrared (NIR) light sensing unit, reflected NIR light;
coordinating, by an image controlling and processing unit, the NIR LED illuminator and the NIR light sensing unit;
analyzing, by an image controlling and processing unit, the reflected NIR light captured by the NIR light sensing unit to generate an image;
determining that a human face exists in the image;
identifying a face region of the human face;
analyzing the face region to extract a feature vector representing the face region;
determining whether a similarity between the feature vector and any of the existing feature vectors in a face feature dictionary is greater than a first threshold; and
when a similarity between the feature vector and a first existing feature vector in the face feature dictionary is greater than the first threshold,
generating a first identification result indicating an identity associated with the first existing feature vector; and
displaying the first identification result;
when the similarity between the feature vector and any of the existing feature vectors in the face feature dictionary is not greater than a first threshold,
generating a second identification result indicating that the feature vector does not exist in the face feature dictionary;
displaying the second identification result; and
storing the face feature in the face feature dictionary.

### Claim 14 (depends on 13)

The method of claim 13, wherein the NIR light sensing unit is a focal plane array (FPA) NIR light sensing unit.

### Claim 15 (depends on 13)

The method of claim 13, wherein the NIR light sensing unit is covered with a color filter, the color filter having a passing band between 825 nm and 875 nm.

### Claim 16 (depends on 13)

The method of claim 13, wherein the NIR light sensing unit is covered with a color filter, the color filter having a passing band between 915 nm and 965 nm.

### Claim 17 (depends on 13)

The method of claim 13, wherein the image controlling and processing unit coordinates the NIR LED illuminator and the NIR light sensing unit by controlling one or more of:
a duty cycle of the NIR LED illuminator;
an analog gain of the NIR light sensing unit;
a digital gain of the NIR light sensing unit;
an exposure time of the NIR light sensing unit; and
a frame rate of the NIR light sensing unit.

### Claim 18 (depends on 17)

The method of claim 17, wherein the image controlling and processing unit coordinates the NIR LED illuminator and the NIR light sensing unit to generate an image with optimal imaging quality.

### Claim 19 (depends on 13)

The method of claim 13, wherein the determining that a human face exists in the image and the identifying a face region of the human face is by:
employing a deep neural network (DNN).

### Claim 20 (depends on 19)

The method of claim 19, wherein the deep neural network is a Multi-Task Convolutional Neural Network (MTCNN).

### Claim 21 (depends on 19)

The method of claim 19, wherein the deep neural network is a Fast Region-based Convolutional Neural Network (Fast R-CNN).

### Claim 22 (depends on 13)

The method of claim 13, further comprising:
calibrating the face region to a calibrated face region associated with an upright pose of a driver, wherein the calibrated face region is analyzed to extract a feature vector representing the calibrated face region.

### Claim 23 (depends on 13)

The method of claim 13, wherein the analyzing the face region to extract a feature vector representing the face region is by employing one or more of:
a backbone network;
a local feature descriptor;
a clustering technique; and
a dimension reduction technique.

### Claim 24 (depends on 13)

The method of claim 13, wherein the similarity is a Cosine similarity.

### Claim 25

A non-transitory tangible computer-readable medium storing instructions which, when executed by one or more processors, cause a method for vehicle driver recognition to be performed, the method comprising:
emitting, by a near infrared (NIR) light-emitting diode (LED) illuminator, NIR light in the vehicle;
capturing, by a near infrared (NIR) light sensing unit, reflected NIR light;
coordinating, by an image controlling and processing unit, the NIR LED illuminator and the NIR light sensing unit;
analyzing, by an image controlling and processing unit, the reflected NIR light captured by the NIR light sensing unit to generate an image;
determining that a human face exists in the image;
identifying a face region of the human face;
analyzing the face region to extract a feature vector representing the face region;
determining whether a similarity between the feature vector and any of the existing feature vectors in a face feature dictionary is greater than a first threshold; and
when a similarity between the feature vector and a first existing feature vector in the face feature dictionary is greater than the first threshold,
generating a first identification result indicating an identity associated with the first existing feature vector; and
displaying the first identification result;
when the similarity between the feature vector and any of the existing feature vectors in the face feature dictionary is not greater than a first threshold,
generating a second identification result indicating that the feature vector does not exist in the face feature dictionary;
displaying the second identification result; and
storing the face feature in the face feature dictionary.
