# US 2016/0253550 A1 · Eye location method and device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0253550 A1
- Document ID: US20160253550A1
- Status: Published application
- Kind: A1
- Application number: 15/030,485
- Filed: 2014-08-14
- Published: 2016-09-01
- Earliest claimed: 2013-11-11
- Assignees: Beijing Techshino Technology Co., Ltd.
- Applicants: Beijing Techshino Technology Co., Ltd.
- Inventors: Qinqin Zhang; Xiangde Zhang; Yaoluo Zhang; Qianying Li
- Web page: https://patentgrep.com/patent/US20160253550A1

## CPC

- G06V 10/42: Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
- G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 10/449: {Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters}
- G06V 10/764: using classification, e.g. of video objects
- G06V 40/165: {using facial parts and geometric relationships}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06V 40/19: Sensors therefor
- G06V 40/193: {Preprocessing; Feature extraction}
- G06T 2207/20024: Filtering details
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30201: Face
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An eye location method and device includes: receiving a face image; locating a position of a nose and positions of eyes in the face image; determining a facial symmetry axis according to the position of the nose; and regulating the positions of the eyes by the facial symmetry axis to obtain target positions of the eyes. By the method and the device, the problem of poorer eye location accuracy in the conventional art is solved, and the effect of improving eye location accuracy is further achieved.

## Claims

### Claim 1

An eye location method, comprising:
receiving a face image;
locating a position of a nose and positions of eyes in the face image, wherein convolution calculation is performed by virtue of a preset nose template image and the face image to locate the nose in the face image;
determining a facial symmetry axis according to the position of the nose; and
regulating the positions of the eyes by the facial symmetry axis to obtain target positions of the eyes.

### Claim 2

(canceled)

### Claim 3 (depends on 1)

The eye location method according to claim 1, wherein the preset nose template image comprises a first template image and a second template image, wherein nostril angles in the first template image are different from nostril angles in the second template image, and performing convolution calculation by a preset nose template image and the face image to locate the nose in the face image comprises:
performing convolution calculation by the face image and the first template image to obtain a first convolution image, performing convolution calculation by the face image and the second template image to obtain a second convolution image;
calculating regional maximums of the first convolution image and the second convolution image respectively;
marking positions with the regional maximum on the first convolution image and positions with the regional maximum on the second convolution image into 1 and marking positions without the regional maximum on the first convolution image and positions without the regional maximum on the second convolution image into 0 to obtain a binary image of the first convolution image and a binary image of the second convolution image;
combining a first binary image and a second binary image to obtain a third binary image, wherein the first binary image is the binary image of the first convolution image and the second binary image is the binary image of the second convolution image; and
determining a regional gravity centre of positions with a regional maximum in the third binary image as the position of the nose.

### Claim 4 (depends on 3)

The eye location method according to claim 3, wherein determining the regional gravity centre further comprises:
dilating the third binary image by a preset matrix to obtain multiple connected regions;
searching a largest connected region in the multiple connected regions; and
determining a gravity centre of the largest connected region as the regional gravity centre.

### Claim 5 (depends on 3)

The eye location method according to claim 3, wherein, after combining a first binary image and a second binary image to obtain a third binary image and before the determining a regional gravity centre of positions with a regional maximum in the third binary image as the position of the nose, comprises performing filtering by a Support Vector Machine (SVM) further comprising:
judging target points by the SVM, wherein the target points are points which are 1 in the third binary image;
under the condition that the target points are judged to be still marked into 1 by the SVM, keeping the target points to be marked into 1 in the third binary image; and
under the condition that the target points are judged to be should be marked into 0 by the SVM, remarking the target points into 0 in the third binary image,
wherein the regional gravity centre of the positions with the regional maximum in the third binary image which is remarked is determined as the position of the nose.

### Claim 6 (depends on 1)

The eye location method according to claim 1, wherein the locating the position of the nose and the positions of the eyes in the face image further comprises:
detecting a first region where the eyes are located in the face image by an AdaBoost algorithm;
performing radial symmetric transformation on the first region to determine opened or closed states of the eyes;
under the condition that the eyes are determined to be in the opened state, locating the positions of the eyes by a Gabor filter; and
under the condition that the eyes are determined to be in the closed state, locating the positions of the eyes by a K-means clustering algorithm.

### Claim 7 (depends on 6)

The eye location method according to claim 6, wherein a polar coordinate of the Gabor filter is G(r,θ)=e−iω(θ−θ0)e−(r−r0)2/α2e−(θ−θ0)2/β2, where r0 and θ0 are parameter factors determining a centre of the Gabor filter, α and β are scale factors of a Gaussian window in the Gabor filter, α and β are determined according to a size of the face image, and ω is an oscillating frequency of the Gabor filter.

### Claim 8 (depends on 6)

The eye location method according to claim 6, wherein determining clustering centres in the K-means clustering algorithm further comprises:
acquiring eye greyscale values in the first region and the number of pixels of each eye greyscale value;
calculating a greyscale value sum according to formula A=aminnmin+amin+1nmin+1+amin+2nmin+2+ . . . amaxnmax, wherein amin is a greyscale minimum in the eye greyscale values, nmin is the number of the pixels of the greyscale minimum, amax is a greyscale maximum in the eye greyscale values, nmax is the number of the pixels of the greyscale maximum, amin+n is a greyscale value which is n more than the greyscale minimum, nmin+n is the number of the pixels of the greyscale value amin+n, and nε[1, 2, 3, . . . , 254]; and
determining the greyscale values which are
1

A
4
,
1

A
2


and


3

A
4
respectively as three clustering centres in the K-means clustering algorithm.

### Claim 9 (depends on 1)

The eye location method according to claim 1, wherein regulating the positions of the eyes by the facial symmetry axis to obtain target positions of the eyes comprises that:
establishing a target triangle by taking a left eye position p1 in the positions of the eyes, a right position p2 in the positions of the eyes and the position p3 of the nose as vertexes;
calculating side lengths Dij of the target triangle according to formula Dij=∥pi−pj∥1/2;
calculating proportions rij among the side lengths of the target triangle according to formula
r
ij
=
D
ik
D
jk
;
establishing for a cosine value cos(αij) of each interior angle of the target triangle according to formula
cos

(
α
ij
)
=
D
ij
2
+
D
ik
2
+
D
jk
2
2

D
ij

D
ik
;
judging whether Pfinal<T is true or not,
wherein
P
final
=
∑
I
≠
J
∈
{
1
,
2
,
3
}

N

(
D
ij
,
μ
ij
,
σ
ij
)
σ
ij
+
∑
i
≠
j
≠
k
∈
{
1
,
2
,
3
}

N

(
D
ij
D
ik
,
μ
ijk
,
σ
ijk
)
σ
ijk
+
∑
i
≠
j
≠
k
∈
{
1
,
2
,
3
}

N

(
D
ij
2
+
D
Ik
2
-
D
jk
2
2

D
ij

D
ik
,
μ
ijk
′
,
σ
ijk
′
)
σ
ijk
′
,
N

(
D
ij
,
μ
ij
,
σ
ij
)
σ
ij
is Gaussian distribution of
D
ij
-
||
p
i
-
p
j

||
1
2
,
N

(
D
ij
D
ik
,
μ
ijk
,
σ
ijk
)
σ
ijk
is Gaussian distribution of
r
ij
=
D
ik
D
jk
,
N

(
D
ij
2
+
D
ik
2
-
D
jk
2
2

D
ij

D
ik
,
μ
ijk
′
,
σ
ijk
′
)
σ
ijk
′
is Gaussian distribution of
cos

(
α
ij
)
=
D
ij
2
+
D
ik
2
+
D
jk
2
2

D
ij

D
ik
,
and T is a preset value;
under the condition that Pfinal<T is determined to be true, determining the left eye position as a symmetry point of the right eye position about the facial symmetry axis, or determining the right eye position as a symmetry point of the left eye position about the facial symmetry axis; and
re-establishing a target triangle until Pfinal≧T is determined.

### Claim 10

An eye location device, comprising:
a receiving unit, configured to receive a face image;
a locating unit, configured to locate a position of a nose and positions of eyes in the face image, wherein the locating unit is configured to perform convolution calculation by a preset nose template image and the face image to locate the nose in the face image;
a first determining unit, configured to determine a facial symmetry axis according to the position of the nose; and
a regulating unit, configured to regulate the positions of the eyes by the facial symmetry axis to obtain target positions of the eyes.

### Claim 11

(canceled)

### Claim 12 (depends on 10)

The eye location device according to claim 10, wherein the preset nose template image comprises a first template image and a second template image, wherein nostril angles in the first template image are different from nostril angles in the second template image, and the locating unit comprises:
a first calculating subunit, configured to perform convolution calculation by the face image and the first template image to obtain a first convolution image, perform convolution calculation by the face image and the second template image to obtain a second convolution image;
a second calculating subunit, configured to calculate regional maximums of the first convolution image and the second convolution image respectively;
a first marking subunit, configured to mark positions with the regional maximum on the first convolution image and positions with the regional maximum on the second convolution image into 1 and marking positions without the regional maximum on the first convolution image and positions without the regional maximum on the second convolution image into 0 to obtain a binary image of the first convolution image and a binary image of the second convolution image;
a combining subunit, configured to combine a first binary image and a second binary image to obtain a third binary image, wherein the first binary image is the binary image of the first convolution image and the second binary image is the binary image of the second convolution image; and
a first determining subunit, configured to determine a regional gravity centre of positions with a regional maximum in the third binary image as the position of the nose.

### Claim 13 (depends on 12)

The eye location device according to claim 12, wherein the first determining subunit further comprises:
a processing unit, configured to dilate the third binary image by a preset matrix to obtain multiple connected regions by a preset matrix;
a searching unit, configured to search a largest connected region in the multiple connected regions; and
a second determining unit, configured to determine a gravity centre of the largest connected region as the regional gravity centre.

### Claim 14 (depends on 12)

The eye location device according to claim 12, wherein the locating unit further comprises:
a first judging subunit, configured to judge target points by the SVM, wherein the target points are points which are 1 in the third binary image;
a keeping subunit, configured to, under the condition that the target points are judged to be still marked into 1 by the SVM, keep the target points to be marked into 1 in the third binary image; and
a second marking subunit, configured to, under the condition that the target points are judged to be should be mark into 0 by the SVM, remark the target points into 0 in the third binary image,
wherein the first determining subunit is configured to determine the regional gravity centre of the positions with the regional maximum in the third binary image which is remarked as the position of the nose.

### Claim 15 (depends on 10)

The eye location device according to claim 10, wherein the locating unit further comprises:
a detecting subunit, configured to detect a first region where the eyes are located in the face image by an AdaBoost algorithm;
a transformating subunit, configured to perform radial symmetric transformation on the first region to determine opened or closed states of the eyes;
a first locating subunit, configured to, under the condition that the eyes are determined to be in the opened state, locate the positions of the eyes by a Gabor filter; and
a second locating subunit, configured to, under the condition that the eyes are determined to be in the closed state, locate the positions of the eyes by a K-means clustering algorithm.

### Claim 16 (depends on 15)

The eye location device according to claim 15, wherein a polar coordinate of the Gabor filter is G(r,θ)=e−iω(θ−θ0)e−(r−r0)2/α2e−(θ−θ0)2/β2, where r0 and θ0, are parameter factors determining a centre of the Gabor filter, α and β are scale factors of a Gaussian window in the Gabor filter, α and β are determined according to a size of the face image, and ω is an oscillating frequency of the Gabor filter.

### Claim 17 (depends on 15)

The eye location device according to claim 15, wherein the second locating subunit further comprises:
an acquiring unit, configured to acquire eye greyscale values in the first region and the number of pixels of each eye greyscale value;
a calculating unit, configured to calculate a greyscale value sum according to formula A=aminnmin+amin+1nmin+1+amin+2nmin+2+ . . . amaxnmax, wherein amin is a greyscale minimum in the eye greyscale values, nmin is the number of the pixels of the greyscale minimum, amax is a greyscale maximum in the eye greyscale values, nmax is the number of the pixels of the greyscale maximum, amin+n is a greyscale value which is n more than the greyscale minimum, nmin+n is the number of the pixels of the greyscale value amin+n, and nε[1, 2, 3, . . . , 254]; and
a third determining unit, configured to determine the greyscale values which are
1

A
4
,
1

A
2


and


3

A
4
respectively as three clustering centres in the K-means clustering algorithm.

### Claim 18 (depends on 10)

The eye location device according to claim 10, wherein the regulating unit comprises:
an establishing subunit, configured to establish a target triangle by taking a left eye position p1 in the positions of the eyes, a right position p2 in the positions of the eyes and the position p3 of the nose as vertexes;
a third calculating subunit, configured to calculate side lengths Dij of the target triangle according to formula Dij=∥pi−pj∥1/2;
a fourth calculating subunit, configured to calculate proportions rij among the side lengths of the target triangle according to formula
r
ij
=
D
ik
D
jk
;
a fifth calculating subunit, configured to calculate a cosine value cos(αij) of each interior angle of the target triangle is calculated according to formula
cos

(
α
ij
)
=
D
ij
2
+
D
ik
2
+
D
jk
2
2

D
ij

D
ik
;
a second judging subunit, configured to judge whether Pfinal<T is true or not, wherein
P
final
=
∑
I
≠
J
∈
{
1
,
2
,
3
}

N

(
D
ij
,
μ
ij
,
σ
ij
)
σ
ij
+
∑
i
≠
j
≠
k
∈
{
1
,
2
,
3
}

N

(
D
ij
D
ik
,
μ
ijk
,
σ
ijk
)
σ
ijk
+
∑
i
≠
j
≠
k
∈
{
1
,
2
,
3
}

N

(
D
ij
2
+
D
Ik
2
-
D
jk
2
2

D
ij

D
ik
,
μ
ijk
′
,
σ
ijk
′
)
σ
ijk
′
,
N

(
D
ij
,
μ
ij
,
σ
ij
)
σ
ij
is Gaussian distribution of
D
ij
-
||
p
i
-
p
j

||
1
2
,
N

(
D
ij
D
ik
,
μ
ijk
,
σ
ijk
)
σ
ijk
is Gaussian distribution of
r
ij
=
D
ik
D
jk
,
N

(
D
ij
2
+
D
ik
2
-
D
jk
2
2

D
ij

D
ik
,
μ
ijk
′
,
σ
ijk
′
)
σ
ijk
′
is Gaussian distribution of
cos

(
α
ij
)
=
D
ij
2
+
D
ik
2
+
D
jk
2
2

D
ij

D
ik
,
and T is a preset value;
a second determining subunit, configured to, under the condition that Pfinal<T is determined to be true, determine the left eye position as a symmetry point of the right eye position about the facial symmetry axis, or determine the right eye position as a symmetry point of the left eye position about the facial symmetry axis,
wherein the establishing subunit is further configured to re-establish a target triangle until the judging subunit judges that Pfinal≧T.
