# US 2024/0203162 A1 · Pupil detection device, operating method of pupil detection device, and computing device including pupil detection device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0203162 A1
- Document ID: US20240203162A1
- Status: Published application
- Kind: A1
- Application number: 18/495,521
- Filed: 2023-10-26
- Published: 2024-06-20
- Earliest claimed: 2022-12-15
- Assignees: Electronics and Telecommunications Research Institute
- Applicants: Electronics and Telecommunications Research Institute
- Inventors: Jang-Hee YOO; Cheolhwan YOO; ByungOk HAN
- Web page: https://patentgrep.com/patent/US20240203162A1

## CPC

- G06V 40/19: Sensors therefor
- G06T 5/70: Denoising; Smoothing
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/82: using neural networks
- G06V 40/193: {Preprocessing; Feature extraction}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Disclosed is a computing device, which includes a processor, a camera that captures RGB image data, a memory that stores image data including the RGB image data, and a pupil detection device that detects a pupil from the RGB image data stored in the memory in response to a request of the processor. The pupil detection device includes an image converter that receives the RGB image data, detects first eye area data including eye area information from the received RGB image data, and converts the first eye area data into second eye area data having IR (InfraRed) image characteristics by using a deep neural network, a pupil candidate detector that detects pupil candidate data from the second eye area data, and a pupil boundary detector that detects pupil boundary data from the pupil candidate data.

## Claims

### Claim 1

A computing device comprising:
a processor;
a camera configured to capture RGB image data;
a memory configured to store image data including the RGB image data; and
a pupil detection device configured to detect a pupil from the RGB image data stored in the memory in response to a request of the processor, and
wherein the pupil detection device includes:
an image converter configured to receive the RGB image data, to detect first eye area data including eye area information from the received RGB image data, and to convert the first eye area data into second eye area data having InfraRed (IR) image characteristics by using a deep neural network;
a pupil candidate detector configured to detect pupil candidate data from the second eye area data; and
a pupil boundary detector configured to detect pupil boundary data from the pupil candidate data.

### Claim 2 (depends on 1)

The computing device of claim 1, wherein the pupil candidate detector includes an image enhancer configured to perform a preprocessing on the second eye area data.

### Claim 3 (depends on 2)

The computing device of claim 2, wherein the preprocessing includes at least one operation of removing noise and adjusting brightness.

### Claim 4 (depends on 1)

The computing device of claim 1, wherein the pupil detection device further includes a gaze calculator configured to calculate gaze data including gaze information from the pupil boundary data.

### Claim 5 (depends on 4)

The computing device of claim 4, wherein the gaze calculator includes:
a pupil center detector configured to detect length information from the pupil boundary data, and to detect pupil center data including center position information of the pupil based on the length information; and
a gaze position calculator configured to calculate gaze position information from the pupil boundary data and the pupil center data, and to calculate the gaze data based on the gaze position information.

### Claim 6 (depends on 5)

The computing device of claim 5, wherein the length information includes long axis information and short axis information of the pupil.

### Claim 7 (depends on 6)

The computing device of claim 6, wherein the gaze position calculator calculates the gaze position information based on a camera coordinate system and a gaze coordinate system.

### Claim 8 (depends on 6)

The computing device of claim 6, wherein the gaze position calculator calculates the gaze position information based on a regression model.

### Claim 9 (depends on 6)

The computing device of claim 6, wherein the pupil detection device further includes a corneal reflection light detector configured to detect corneal reflection light data including the eye area information from corneal reflection light by illumination.

### Claim 10 (depends on 9)

The computing device of claim 9, wherein the corneal reflection light detector includes a corneal reflection light center detector configured to detect center information of the corneal reflection light, and
the corneal reflection data includes the center information of the corneal reflection light.

### Claim 11 (depends on 10)

The computing device of claim 10, wherein the corneal reflection light detector compares the pupil candidate data with the corneal reflection light data to verify the pupil candidate data.

### Claim 12 (depends on 11)

The computing device of claim 11, wherein the pupil boundary detector further includes:
a pupil detector configured to detect pupil area information including area information of the pupil based on the comparison result; and
an ellipse boundary detector configured to detect the pupil boundary data using a boundary detection algorithm based on the pupil area information.

### Claim 13

A method of operating a pupil detection device, the method comprising:
receiving RGB image data;
detecting first eye area data including eye area information from the RGB image data;
converting the first eye area data into second eye area data having infrared image characteristics using a deep neural network;
detecting pupil candidate data from the second eye area data; and
detecting pupil boundary data from the pupil candidate data.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
calculating gaze data including gaze information from the pupil boundary data.

### Claim 15 (depends on 14)

The method of claim 14, wherein the calculating of the gaze data including the gaze information from the pupil boundary data includes:
detecting length information from the pupil boundary data;
detecting pupil center data including center position information of the pupil based on the length information;
calculating gaze position information from the pupil boundary data and the pupil center data; and
calculating the gaze data based on the gaze position information.

### Claim 16 (depends on 15)

The method of claim 15, wherein the length information includes long axis information and short axis information of the pupil.

### Claim 17 (depends on 15)

The method of claim 15, further comprising:
after detecting the pupil candidate data, verifying the pupil candidate data.

### Claim 18 (depends on 17)

The method of claim 17, wherein the verifying of the pupil candidate data includes:
detecting corneal reflection light data from corneal reflection light by illumination; and
comparing the pupil candidate data with the corneal reflection data.

### Claim 19 (depends on 18)

The method of claim 18, wherein the detecting of the pupil boundary data from the pupil candidate data includes detecting the pupil boundary data based on the comparison result.

### Claim 20 (depends on 19)

The method of claim 19, wherein the detecting of the pupil boundary data based on the comparison result includes:
detecting pupil area data including area information of the pupil based on the comparison result; and
detecting the pupil boundary data from the pupil area data based on a boundary detection algorithm.
