# US 11,992,317 B2 · Alertness estimation apparatus, alertness estimation method, and computer- readable recording medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,992,317 B2
- Document ID: US11992317B2
- Status: Granted
- Kind: B2
- Application number: 17/425,840
- Filed: 2019-02-01
- Published: 2022-10-06
- Granted: 2024-05-28
- Earliest claimed: 2019-02-01
- Assignees: NEC Corporation
- Applicants: NEC Corporation
- Inventors: Masanori Tsujikawa
- Web page: https://patentgrep.com/patent/US11992317B2

## CPC

- A61B 5/18: for vehicle drivers {or machine operators}
- A61B 5/02416: {using photoplethysmograph signals, e.g. generated by infrared radiation (A61B5/14552 takes precedence)}
- A61B 5/163: {by tracking eye movement, gaze, or pupil change}
- A61B 5/4809: {Sleep detection, i.e. determining whether a subject is asleep or not}
- A61B 5/7203: {for noise prevention, reduction or removal}
- A61B 5/7267: {involving training the classification device}
- G06V 10/60: relating to illumination properties, e.g. using a reflectance or lighting model
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06V 10/82: using neural networks
- G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06V 40/18: Eye characteristics, e.g. of the iris
- G06V 40/193: {Preprocessing; Feature extraction}
- G08B 21/06: indicating a condition of sleep, e.g. anti-dozing alarms
- A61B 5/0077: {Devices for viewing the surface of the body, e.g. camera, magnifying lens}
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS

## Abstract

An alertness estimation apparatus 10, includes: an image extraction unit 11 that extracts an image of a portion including eyes from an image data of a face image of a human; a first alertness estimation unit 12 that estimates a first alertness based on the extracted image; a pulse wave detection unit 13 that determines whether or not the estimated first alertness satisfies a set condition, and detect a pulse wave of the human from the face image, when the first alertness satisfies the set condition; and a second alertness estimation unit 14 that estimates a second alertness of the human based on the detected pulse wave.

## Claims

### Claim 1

An alertness estimation apparatus comprising:
at least one processor; and
at least one memory storing instructions executable by the at least one processor to:
extract an eyes image including eyes of a person from a face image including a face of the person;
calculate a degree of opening and closing of eyelids of the eyes of the person in the eyes image;
calculate a length of time during which the eyelids are closed from a time-series change in the degree of opening and closing of the eyelids;
determine that the person is awake when the length of time is equal to or less than a threshold value;
determine that the person is sleeping when the length of time exceeds the threshold value;
set a first alertness level to one when the person has been determined to be awake;
set the first alertness level to zero when the person has been determined to be sleeping;
detect a pulse wave of the human person from the face image when the first alertness level has been set to zero; and
estimate a second alertness level of the person based on the pulse wave.

### Claim 2 (depends on 1)

The alertness estimation apparatus according to claim 1, wherein
the pulse wave of the person is detected by detecting a time-series change of a brightness value of a specific pixel in the face image,
the instructions are executable by the at least one processor to again determine whether the person is awake or sleeping, according to the time-series change of the brightness of the specific pixel,
the second alertness level is set to one when the person has been determined to be awake according to the time-series change of the brightness of the specific pixel, and
the second alertness level is set to zero when the person has been determined to be sleeping according to the time-series change of the brightness of the specific pixel.

### Claim 3 (depends on 2)

The alertness estimation apparatus according to claim 2, wherein
whether the human is awake or sleep is determined according to the time-series change of the brightness of the specific pixel by using a learning model constructed by machine learning a relationship between a human behavior human pulse wave.

### Claim 4 (depends on 2)

The alertness estimation apparatus according to claim 2, wherein
whether the human is awake or sleep is determined according to the time-series change of the brightness of the specific pixel by using a learning model constructed by machine learning a relationship between human brain activity and human pulse.

### Claim 5 (depends on 1)

The alertness estimation apparatus according to claim 1, wherein
the instructions are executable by the at least one processor to further remove noise contained in the pulse wave according to a heart rate estimated from the pulse wave, and
the second alertness is estimated after the noise contained in the pulse wave has been removed.

### Claim 6

An alertness estimation method comprising:
extracting, by a processor, an eyes image including eyes of a person from a face image including a face of the person;
calculating, by the processor, a degree of opening and closing of eyelids of the eyes of the person in the eyes image;
calculating, by the processor, a length of time during which the eyelids are closed from a time-series change in the degree of opening and closing of the eyelids;
determining, by the processor, that the person is awake when the length of time is equal to or less than a threshold value;
determining, by the processor, that the person is sleeping when the length of time exceeds the threshold value;
setting, by the processor, a first alertness level to one when the person has been determined to be awake;
setting, by the processor, the first alertness level to zero when the person has been determined to be sleeping;
detecting, by the processor, a pulse wave of the person from the face image when the first alertness level has been set to zero; and
estimating, by the processor, a second alertness level of the person based on the pulse wave.

### Claim 7 (depends on 6)

The alertness estimation method according to claim 6, wherein
the pulse wave of the person is detected by detecting a time-series change of a brightness value of a specific pixel in the face image,
the method further comprises again determining, by the processor, whether the person is awake or sleeping, according to the time-series change of the brightness of the specific pixel,
the second alertness level is set to one when the person has been determined to be awake according to the time-series change of the brightness of the specific pixel, and
the second alertness level is set to zero when the person has been determined to be sleeping according to the time-series change of the brightness of the specific pixel.

### Claim 8 (depends on 7)

The alertness estimation method according to claim 7, wherein
whether the human is awake or sleep is determined according to the time-series change of the brightness of the specific pixel by using a learning model constructed by machine learning a relationship between a human behavior human pulse wave.

### Claim 9 (depends on 7)

The alertness estimation method according to claim 7, wherein
whether the human is awake or sleep is determined according to the time-series change of the brightness of the specific pixel by using a learning model constructed by machine learning a relationship between human brain activity and human pulse.

### Claim 10 (depends on 6)

The alertness estimation method according to claim 6, wherein
the method further comprises removing, by the processor, noise contained in the pulse wave according to a heart rate estimated from the pulse wave, and
the second alertness is estimated after the noise contained in the pulse wave has been removed.

### Claim 11

A non-transitory computer-readable recording medium storing a program executable by a computer to carry out:
extracting an eyes image including eyes of a person from a face image including a face of the person;
calculating a degree of opening and closing of eyelids of the eyes of the person in the eyes image;
calculating a length of time during which the eyelids are closed from a time-series change in the degree of opening and closing of the eyelids;
determining that the person is awake when the length of time is equal to or less than a threshold value;
determining that the person is sleeping when the length of time exceeds the threshold value;
setting a first alertness level to one when the person has been determined to be awake;
setting the first alertness level to zero when the person has been determined to be sleeping;
detecting a pulse wave of the person from the face image when the first alertness level has been set to zero; and
estimating a second alertness level of the person based on the pulse wave.

### Claim 12 (depends on 11)

The non-transitory computer-readable recording medium according to claim 11, wherein
the pulse wave of the person is detected by detecting a time-series change of a brightness value of a specific pixel in the face image,
the program is executable by the computer to further carry out again determining whether the person is awake or sleeping, according to the time-series change of the brightness of the specific pixel,
the second alertness level is set to one when the person has been determined to be awake according to the time-series change of the brightness of the specific pixel, and
the second alertness level is set to zero when the person has been determined to be sleeping according to the time-series change of the brightness of the specific pixel.

### Claim 13 (depends on 12)

The non-transitory computer-readable recording medium according to claim 12, wherein
whether the human is awake or sleep is determined according to the time-series change of the brightness of the specific pixel by using a learning model constructed by machine learning a relationship between a human behavior human pulse wave.

### Claim 14 (depends on 12)

The non-transitory computer-readable recording medium according to claim 12, wherein
whether the human is awake or sleep is determined according to the time-series change of the brightness of the specific pixel by using a learning model constructed by machine learning a relationship between human brain activity and human pulse.