# US 2017/0286771 A1 · Gaze detection apparatus and gaze detection method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0286771 A1
- Document ID: US20170286771A1
- Status: Published application
- Kind: A1
- Application number: 15/448,722
- Filed: 2017-03-03
- Published: 2017-10-05
- Earliest claimed: 2016-03-31
- Assignees: Fujitsu Limited
- Applicants: Fujitsu Limited
- Inventors: Daisuke Ishii; Satoshi NAKASHIMA
- Web page: https://patentgrep.com/patent/US20170286771A1

## CPC

- A61B 3/113: for determining or recording eye movement
- A61B 3/0025: {characterised by electronic signal processing, e.g. eye models}
- G06T 7/73: using feature-based methods
- G06V 40/165: {using facial parts and geometric relationships}
- G06V 40/19: Sensors therefor
- G06V 40/193: {Preprocessing; Feature extraction}
- G06V 40/197: {Matching; Classification}
- G06T 2207/30201: Face
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A gaze detection apparatus includes: an image capturing unit which generates an image representing both eyes of a user; an eye detection unit which, for each of the eyes, detects an eye region containing the eye from the image; a pupil detection unit which, for one of the eyes, detects a pupil region containing a pupil from the eye region; an estimating unit which obtains, based on the pupil region detected for the one eye, an estimated shape of the pupil region on the image for the other one of the eyes; a redetection unit which detects the pupil region by searching for a region having a shape that matches the estimated shape from within the eye region detected for the other eye; and a gaze detection unit which detects the user's gaze direction based on a position of the pupil region detected for each of the eyes.

## Claims

### Claim 1

A gaze detection apparatus comprising:
a processor configured to:
detect, for each of the eyes, an eye region containing the eye from an image representing both eyes of u user;
detect, for one of the eyes, a pupil region containing a pupil from the eye region;
obtain, based on the pupil region detected for the one eye, an estimated shape of the pupil region on the image for the other one of the eyes;
detect the pupil region by searching for a region having a shape that matches the estimated shape from within the eye region detected for the other eye; and
detect the user's gaze direction based on a position of the pupil region detected for each of the eyes.

### Claim 2 (depends on 1)

The gaze detection apparatus according to claim 1, wherein the processor is further configured to set, when a first angle which the gaze direction detected for the one eye makes with a line joining the one eye to the camera is smaller than a second angle which the gaze direction estimated for the other eye makes with a line joining the other eye to the camera, the ratio of a width of the estimated shape of the pupil region of the other eye, measured in a first direction along which the eyes are arranged, to the width thereof measured in a second direction perpendicular to the first direction, smaller than the ratio of the width of the pupil region of the one eye, measured in the first direction, to the width thereof measured in the second direction, while on the other hand, the processor sets, when the first angle is larger than the second angle, the ratio of the width of the estimated shape of the pupil region of the other eye, measured in the first direction, to the width thereof measured in the second direction, larger than the ratio of the width of the pupil region of the one eye, measured in the first direction, to the width thereof measured in the second direction.

### Claim 3 (depends on 1)

The gaze detection apparatus according to claim 1, wherein the processor is further configured to:
detect, for each of the eyes, a reference point whose position does not change with the user's gaze direction; and
obtain, based on a positional relationship between the pupil region and the reference point detected for the one eye and on the reference point detected for the other eye, an estimated range within which the pupil region for the other eye is estimated to be located, and wherein the processor detects the pupil region by searching for a region having a shape that matches the estimated shape from within the estimated range.

### Claim 4 (depends on 3)

The gaze detection apparatus according to claim 3, further comprising an illuminating light source configured to illuminate both of the eyes, and wherein
the processor detects a position of a corneal reflection image of the illuminating light source as the reference point for each of the eyes, estimates, based on the positional relationship between the pupil region and the reference point detected for the one eye, a positional relationship between the gaze direction and the illuminating light source, as viewed in a first direction along which the eyes are arranged, and obtains the estimated range based on the estimated positional relationship.

### Claim 5 (depends on 4)

The gaze detection apparatus according to claim 4, wherein the processor sets, when the positional relationship of the gaze direction relative to the illuminating light source as viewed in the first direction is identical to the positional relationship of the other eye relative to the one eye as viewed in the first direction, the estimated range so that a distance from the reference point for the other eye to a center of the estimated range is longer than a distance from the reference point for the one eye to a centroid of the pupil region along a direction directed away from the reference point for the one eye and toward the pupil region.

### Claim 6 (depends on 3)

The gaze detection apparatus according to claim 3, wherein the processor detects, for each of the eyes, a position of the eye's inner corner as the reference point, and
sets the estimated range so that a distance from the reference point for the other eye to a center of the estimated range is shorter as a distance from the reference point for the one eye to a centroid of the pupil region is longer.

### Claim 7 (depends on 3)

The gaze detection apparatus according to claim 3, wherein the processor detects, for each of the eyes, a midpoint between the eye's inner and outer corners as the reference point, and
sets, when the positional relationship of the pupil region relative to the reference point for the one eye, as viewed in a first direction along which the eyes are arranged, is identical to the positional relationship of the other eye relative to the one eye, the estimated range so that a distance from the reference point for the other eye to a center of the estimated range is shorter than a distance from the reference point for the one eye to a centroid of the pupil region.

### Claim 8 (depends on 1)

The gaze detection apparatus according to claim 1, wherein the processor detects the pupil region from the eye region for each of the eyes, and wherein the processor is further configured to:
calculate an evaluation score indicating degree of certainty of the detected pupil region; and
determine that, of the two eyes, the eye for which the evaluation score is lower corresponds to the other eye.

### Claim 9

A gaze detection method comprising:
generating an image representing both eyes of a user;
detecting, for each of the eyes, an eye region containing the eye from the image;
detecting, for one of the eyes, a pupil region containing a pupil from the eye region;
obtaining, based on the pupil region detected for the one eye, an estimated shape of the pupil region on the image for the other one of the eyes;
detecting the pupil region by searching for a region having a shape that matches the estimated shape from within the eye region detected for the other eye; and
detecting the user's gaze direction based on a position of the pupil region detected for each of the eyes.

### Claim 10 (depends on 9)

The gaze detection method according to claim 9, further comprising:
setting, when a first angle which the gaze direction detected for the one eye makes with a line joining the one eye to the camera is smaller than a second angle which the gaze direction estimated for the other eye makes with a line joining the other eye to the camera, the ratio of a width of the estimated shape of the pupil region of the other eye, measured in a first direction along which the eyes are arranged, to the width thereof measured in a second direction perpendicular to the first direction, smaller than the ratio of the width of the pupil region of the one eye, measured in the first direction, to the width thereof measured in the second direction, while on the other hand, setting, when the first angle is larger than the second angle, the ratio of the width of the estimated shape of the pupil region of the other eye, measured in the first direction, to the width thereof measured in the second direction, larger than the ratio of the width of the pupil region of the one eye, measured in the first direction, to the width thereof measured in the second direction.

### Claim 11 (depends on 9)

The gaze detection method according to claim 9, further comprising:
detecting, for each of the eyes, a reference point whose position does not change with the user's gaze direction; and
obtaining, based on a positional relationship between the pupil region and the reference point detected for the one eye and on the reference point detected for the other eye, an estimated range within which the pupil region for the other eye is estimated to be located, and wherein detection of the pupil region for the other eye includes detecting the pupil region by searching for a region having a shape that matches the estimated shape from within the estimated range.

### Claim 12 (depends on 11)

The gaze detection method according to claim 11, further comprising illuminating both of the eyes by an illuminating light source, and wherein
detection of the reference point includes detecting a position of a corneal reflection image of the illuminating light source as the reference point for each of the eyes, and
obtaining the estimated range includes estimating, based on the positional relationship between the pupil region and the reference point detected for the one eye, a positional relationship between the gaze direction and the illuminating light source, as viewed in a first direction along which the eyes are arranged, and obtaining the estimated range based on the estimated positional relationship.

### Claim 13 (depends on 12)

The gaze detection method according to claim 12, wherein obtaining the estimated range includes setting, when the positional relationship of the gaze direction relative to the illuminating light source as viewed in the first direction is identical to the positional relationship of the other eye relative to the one eye as viewed in the first direction, the estimated range so that a distance from the reference point for the other eye to a center of the estimated range is longer than a distance from the reference point for the one eye to a centroid of the pupil region along a direction directed away from the reference point for the one eye and toward the pupil region.

### Claim 14 (depends on 11)

The gaze detection method according to claim 11, wherein detection of the reference point includes detecting, for each of the eyes, a position of the eye's inner corner as the reference point, and
obtaining the estimated range includes setting the estimated range so that a distance from the reference point for the other eye to a center of the estimated range is shorter as a distance from the reference point for the one eye to a centroid of the pupil region is longer.

### Claim 15 (depends on 11)

The gaze detection method according to claim 11, wherein detection of the reference point includes detecting, for each of the eyes, a midpoint between the eye's inner and outer corners as the reference point, and
obtaining the estimated range includes setting, when the positional relationship of the pupil region relative to the reference point for the one eye, as viewed in a first direction along which the eyes are arranged, is identical to the positional relationship of the other eye relative to the one eye, the estimated range so that a distance from the reference point for the other eye to a center of the estimated range is shorter than a distance from the reference point for the one eye to a centroid of the pupil region.

### Claim 16 (depends on 9)

The gaze detection method according to claim 9, wherein detection of the pupil region includes detecting the pupil region from the eye region for each of the eyes, and further comprising:
calculating an evaluation score indicating degree of certainty of the detected pupil region; and
determining that, of the two eyes, the eye for which the evaluation score is lower corresponds to the other eye.

### Claim 17

A non-transitory computer-readable recording medium having recorded thereon a gaze detection computer program that causes a computer to execute a process comprising:
detecting, from an image representing both eyes of a user, for each of the eyes, an eye region containing the eye;
detecting, for one of the eyes, a pupil region containing a pupil from the eye region;
obtaining, based on the pupil region detected for the one eye, an estimated shape of the pupil region on the image for the other one of the eyes;
detecting the pupil region by searching for a region having a shape that matches the estimated shape from within the eye region detected for the other eye; and
detecting the user's gaze direction based on a position of the pupil region detected for each of the eyes.
