# US 2019/0172222 A1 · Line-of-sight detection device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0172222 A1
- Document ID: US20190172222A1
- Status: Published application
- Kind: A1
- Application number: 16/324,583
- Filed: 2017-08-10
- Published: 2019-06-06
- Earliest claimed: 2016-08-12
- Assignees: National University Corporation Shizuoka University
- Applicants: National University Corporation Shizuoka University
- Inventors: Yoshinobu EBISAWA
- Web page: https://patentgrep.com/patent/US20190172222A1

## CPC

- G06T 7/73: using feature-based methods
- G06T 7/74: {involving reference images or patches}
- A61B 3/113: for determining or recording eye movement
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06T 7/593: from stereo images
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T 2207/30201: Face
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A line-of-sight detection device 1 includes a camera 10, a light source 13 that causes corneal reflection, and an image processing device 20 that obtains information on a pupil shape of an eye, a position (PS) of a pupil center, and a position (GS) of the cornea reflection and obtains a line of sight of a subject M. The image processing device 20 includes a first optical axis acquisition unit 28 that acquires a first optical axis candidate using the position (PS) of the pupil center and the position (GS) of the cornea reflection, a second optical axis acquisition unit 29 that obtains a second optical axis candidate using the pupil shape, an optical axis information acquisition unit 31 that generates information for acquiring the optical axis, a calibration information unit 33 that provides calibration information, and a visual axis acquisition unit 32 that obtains a visual axis AS as a line of sight by calibrating an optical axis AL using the calibration information.

## Claims

### Claim 1

A line-of-sight detection device comprising:
an image acquisition unit that obtains a plurality of eye images including eyes of a subject;
a light source that is disposed near the image acquisition unit and irradiates the subject with light for causing corneal reflection in the eye; and
an image processing unit that acquires information on a pupil shape of the eye, a position (PS) of a pupil center and a position (GS) of corneal reflection included in the plurality of eye images using the plurality of eye images, and obtains a line of sight of the subject based on an acquired information,
wherein the image processing unit includes
a first optical axis acquisition unit that acquires a first optical axis candidate using the position (PS) of the pupil center and the position (GS) of the corneal reflection;
a second optical axis acquisition unit that acquires a second optical axis candidate by using the pupil shape;
an optical axis information acquisition unit that generates information for acquiring an optical axis using at least one of the first optical axis candidate and the second optical axis candidate on the basis of the pupil shape;
a calibration information unit that provides calibration information indicating a deviation of the optical axis with respect to a visual axis of the eye; and
a visual axis acquisition unit that obtains the visual axis as the line of sight by calibrating the optical axis using the calibration information.

### Claim 2 (depends on 1)

The line-of-sight detection device according to claim 1,
wherein the calibration information is a deviation angle (θ0), and
the deviation angle (θ0) is a vector indicating a deviation of the optical axis with respect to the visual axis, and is indicated by a magnitude (|θ0|) and a slope (ϕ0) of the optical axis with respect to the visual axis.

### Claim 3 (depends on 1)

The line-of-sight detection device according to claim 1,
wherein the image processing unit includes
a first computation unit that obtains a first vector (OP) directed from a position (OS) of the image acquisition unit to the position (PS) of the pupil center;
a second computation unit that obtains a second vector (r) directed from the position (GS) of the corneal reflection to the position (PS) of the pupil center;
a third computation unit that obtains a virtual viewpoint plane having the first vector (OP) as a normal;
a fourth computation unit that approximates an outer shape of the pupil included in the plurality of eye images as an ellipse and obtains an ellipticity (R=LA/LB) indicated by a major diameter (LA) and a minor diameter (LB) of the ellipse; and
a fifth computation unit that obtains a slope (γ′) between a minor axis of the ellipse and a horizontal axis,
the first optical axis acquisition unit acquires the first optical axis candidate indicated by a magnitude (|θ|) of an angle (θ) with reference to the first vector (OP) and a slope (ϕ) with reference to a horizontal axis included in the virtual viewpoint plane using a magnitude (|r|) of the second vector (r) associated with the magnitude (|θ|) of the angle (θ) and a slope (ϕ′) formed by the second vector (r) associated with the slope (ϕ) and the horizontal axis, and
the second optical axis acquisition unit obtains the second optical axis candidate indicated by the magnitude (|θ|) of the angle (θ) with reference to the first vector (OP) and the slope (ϕ) with reference to the horizontal axis included in the virtual viewpoint plane using the ellipticity (R) associated with the magnitude (|θ|) of the angle (θ) and the slope (γ′) associated with the slope (ϕ).

### Claim 4 (depends on 3)

The line-of-sight detection device according to claim 3, wherein the first optical axis acquisition unit associates the magnitude (|r|) of the second vector (r) with the magnitude (|θ|) of the angle (θ) using a first coefficient (k).

### Claim 5 (depends on 4)

The line-of-sight detection device according to claim 4,
wherein the image acquisition unit includes a first camera that obtains a first eye image included in the plurality of eye images and that is disposed at a first camera position (OS1), and a second camera that obtains a second eye image included in the plurality of eye images and that is disposed at a second camera position (OS2),
the eye includes a first eye and a second eye,
the first eye includes a position (GS1) of first corneal reflection, a position (PS1) of a first pupil center, and a first optical axis,
the second eye includes a position (GS2) of second corneal reflection, a position (PS2) of a second pupil center, and a second optical axis,
the image processing unit further includes
a sixth computation unit that obtains information on a first virtual optical axis plane including the first camera position (OS1), the position (PS1) of the first pupil center, and a position at which the first optical axis intersects a first virtual viewpoint plane having a fourth vector as a normal on the basis of a third vector (r1) directed from the position (GS1) of the first corneal reflection to the position (PS1) of the first pupil center and the fourth vector from the first camera position (OS1) to the position (PS1) of the first pupil center using the first eye image;
a seventh computation unit that obtains information on a second virtual optical axis plane including the second camera position (OS2), the position (PS2) of the second pupil center, and a position at which the second optical axis intersects a second virtual viewpoint plane having a sixth vector as a normal on the basis of a fifth vector (r2) directed from the position (GS2) of the second corneal reflection to the position (PS2) of the second pupil center and the sixth vector from the second camera position (OS2) to the position (PS2) of the second pupil center using the second eye image; and
a coefficient unit that obtains the first coefficient (k), and
the coefficient unit obtains the first coefficient (k1) corresponding to the first eye using a fifth angle (θ1) obtained from an intersection line at which the first virtual optical axis plane and the second virtual optical axis plane intersect each other and a magnitude (|r1|) of the third vector (r1), and obtains the first coefficient (k2) corresponding to the second eye using a fifth angle (θ2) obtained from the intersection line and the sixth vector, and a magnitude (|r2|) of the fifth vector (r2).

### Claim 6 (depends on 5)

The line-of-sight detection device according to claim 5,
wherein the fourth computation unit obtains a first ellipse (R1=LA1/LB1) that is a ratio of a major diameter (LA1) to a minor diameter (LB1) of the ellipse indicating an outer shape of the pupil included in the first eye image using the first eye image, and acquires a second ellipse (R2=LA2/LB2) which is a ratio of a major diameter (LA2) to a minor diameter (LB2) of the ellipse indicating an outer shape of the pupil included in the second eye image using the second eye image, and
the image processing unit further includes a direction determination unit that compares the first ellipse (R1) with the second ellipse (R2) to distinguish a line-of-sight direction of the subject in a horizontal direction.

### Claim 7 (depends on 3)

The line-of-sight detection device according to claim 3, wherein the second optical axis acquisition unit associates an inverse cosine component of a reciprocal of the ellipticity (R) with the magnitude (|θ|) of the angle (θ) using a second coefficient (h).

### Claim 8 (depends on 3)

The line-of-sight detection device according to claim 3, wherein the second optical axis acquisition unit uses a corrected ellipticity (R′) obtained by correcting the deviation of a pupil contour caused by refraction of a cornea of an eyeball of the subject in acquisition of the magnitude (|θ|) of the angle (θ).

### Claim 9 (depends on 1)

The line-of-sight detection device according to claim 1,
wherein the eye includes a first eye and a second eye, and
the image processing unit further includes a coordinate acquisition unit that acquires respective three-dimensional coordinates of a position (PS1) of the pupil center in the first eye, a position (PS2) of the pupil center in the second eye, a position (NS1) of a first nostril among a pair of nostrils of the subject, and a position (NS2) of a second nostril among the pair of nostrils of the subject; and
a face posture acquisition unit that acquires a face posture of the subject using the position (PS1) of the pupil center, the position (PS2) of the pupil center, the position (NS1) of the first nostril, and the position (NS2) of the second nostril.

### Claim 10 (depends on 9)

The line-of-sight detection device according to claim 9,
wherein the image acquisition unit include a first camera that obtains a first image including a face of the subject, and a second camera that is disposed at a position separated from the first camera and obtains a second image including the face of the subject, and
the coordinate acquisition unit acquires the respective three-dimensional coordinates of the position (PS1) of the pupil center in the first eye, the position (PS2) of the pupil center in the second eye, the position (NS1) of the first nostril, and the position (NS2) of the second nostril using a stereo matching method based on the first image and the second image.

### Claim 11 (depends on 10)

The line-of-sight detection device according to claim 10,
wherein the image acquisition unit includes one third camera that obtains a third image including a face of the subject, and
the coordinate acquisition unit includes
a distance computation unit that calculates a distance between three feature points generated from the first eye, the second eye, the first nostril, and the second nostril;
a two-dimensional coordinate computation unit that calculates two-dimensional coordinates of three feature points in the third image using the third image acquired by the third camera; and
a three-dimensional coordinate computation unit that calculates three-dimensional coordinates of each of the three generated feature points using the distance acquired by the distance computation unit and the two-dimensional coordinates acquired by the coordinate acquisition unit.

### Claim 12 (depends on 1)

The line-of-sight detection device according to claim 1, wherein the optical axis information acquisition unit generates information for selecting one of the first optical axis candidate and the second optical axis candidate as the optical axis on the basis of the pupil shape.