# US 2021/0241487 A1 · Pupil positioning method and apparatus, vr/ar apparatus and computer readable medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241487 A1
- Document ID: US20210241487A1
- Status: Published application
- Kind: A1
- Application number: 16/956,747
- Filed: 2019-11-19
- Published: 2021-08-05
- Earliest claimed: 2019-01-31
- Assignees: Beijing Boe Optoelectronics Technology Co., Ltd.; Boe Technology Group Co., Ltd.
- Applicants: Beijing Boe Optoelectronics Technology Co., Ltd.; Boe Technology Group Co., Ltd.
- Inventors: Jiankang Sun; Hao Zhang; Lili Chen; Hongzhen Xue; Fuqiang Ma; Minglei Chu; Xi Li
- Web page: https://patentgrep.com/patent/US20210241487A1

## CPC

- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06T 7/13: Edge detection
- G06T 7/136: involving thresholding
- G06T 7/60: Analysis of geometric attributes
- G06T 7/73: using feature-based methods
- G06V 10/754: {involving a deformation of the sample pattern or of the reference pattern; Elastic matching}
- G06V 40/19: Sensors therefor
- G06V 40/193: {Preprocessing; Feature extraction}
- H04N 23/56: provided with illuminating means
- G06T 2207/10048: Infrared image
- G06T 2207/30041: Eye; Retina; Ophthalmic
- G06T 2207/30201: Face
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

The present disclosure relates to a pupil positioning method. The pupil positioning method may include: obtaining an eye image under illumination of a light source; determining a first internal point in a pupil of the eye image; calculating gradient changes of pixel points along a straight line starting from the first internal point toward outside of the pupil; determining a plurality of edge points at an edge of the pupil based on the gradient changes of the pixel points along the straight line; and performing ellipse fitting on the edge points to obtain a pupil center.

## Claims

### Claim 1

A pupil positioning method, comprising:
obtaining an eye image under illumination of a light source;
determining a first internal point in a pupil of the eye image;
calculating gradient changes of pixel points along a straight line starting from the first internal point toward outside of the pupil;
determining a plurality of edge points at an edge of the pupil based on the gradient changes of the pixel points along the straight line; and
performing ellipse fitting on the edge points to obtain a pupil center.

### Claim 2 (depends on 1)

The pupil positioning method according to claim 1, wherein determining the first internal point in the pupil of the eye image comprises:
obtaining imaging points of the light source on a cornea and calculating an average centroid position of all the imaging points in the eye image;
obtaining a pupil scan area based on the average centroid position;
scanning the eye image in the pupil scan area using a pupil scan frame having a predetermined side length according to a predetermined scan step size; and
determining a center of the pupil scan frame having a smallest average gray value as the first internal point.

### Claim 3 (depends on 2)

The pupil positioning method according to claim 2, wherein the pupil scan frame has a side length of d/4 and a scan step size of d/2, a value of d is 1/16 of a width of the eye image or a length of a long axis of a pupil ellipse calculated based on a previous frame of the eye image.

### Claim 4 (depends on 1)

The pupil positioning method according to claim 1, wherein determining the plurality of edge points at the edge of the pupil based on the gradient changes of the pixel points along the straight line comprises:
determining a pixel point, among the pixel points along the straight line, having a gradient change that satisfies a preset condition as a first edge point at the edge of the pupil;
searching for a plurality of pixel points in the image having the same or similar gradient change as the gradient change at the first edge point, and
setting the plurality of pixel points as the second edge point to the Nth edge point, where N is a positive integer greater than 4.

### Claim 5 (depends on 4)

The pupil positioning method according to claim 4, wherein the gradient change is a change in a pixel gray value, and the preset condition is that the change in the pixel gray value at the pixel point is not less than a preset edge gradient.

### Claim 6 (depends on 4)

The pupil positioning method according to claim 4, wherein searching for the plurality of pixel points in the image having the same or similar gradient change as the gradient change at the first edge point comprises:
searching, starting from the first edge point, in neighborhood pixels along a predetermined direction for a first point that a difference between a gradient change at the first point and the gradient change at the first edge point is less than a predetermined threshold, and setting the first point as the second edge point;
searching, starting from the second edge point, in neighborhood pixels along the predetermined direction for a second point that a difference between a gradient change at the second point and the gradient change at the first edge point is less than the predetermined threshold, and setting the second point as the third edge point; and
repeating the above steps until the Nth edge point is obtained.

### Claim 7 (depends on 6)

The pupil positioning method according to claim 6, wherein performing ellipse fitting on the edge points comprises:
dividing the edge points from the first edge point to the Nth edge point into a plurality of consecutive groups;
performing ellipse fitting on the edge points in each of the consecutive groups separately to obtain a plurality of ellipses;
obtaining intersection points of the plurality of ellipses; and
obtaining the pupil center based on the intersection points.

### Claim 8 (depends on 7)

The pupil positioning method according to claim 7 wherein each of the plurality of consecutive groups comprises a substantially same number of edge points.

### Claim 9 (depends on 7)

The pupil positioning method according to claim 7, wherein dividing the edge points from the first edge point to the Nth edge point into the plurality of consecutive groups comprises:
dividing the edge points from the first edge point to the Nth edge point into four groups in horizontal and vertical directions; or
dividing the edge points from the first edge point to the Nth edge point into four groups in a clockwise or counterclockwise direction.

### Claim 10 (depends on 7)

The pupil positioning method according to claim 7, wherein obtaining the pupil center based on the intersection points comprises:
using average values of coordinates of the intersection points as position coordinates of the pupil center.

### Claim 11 (depends on 1)

The pupil positioning method according to claim 1, wherein the light source is an infrared light-emitting diode (LED) lamp and a wavelength of an infrared light from the infrared LED lamp is about 850 nm.

### Claim 12 (depends on 1)

The pupil positioning method according to claim 1, wherein the eye image is captured by an infrared camera.

### Claim 13 (depends on 1)

The pupil positioning method according to claim 1, before determining the first internal point in the pupil of the eye image, further comprising preprocessing the eye image including gray value conversion, histogram equalization, and Gaussian filtering.

### Claim 14

A pupil positioning apparatus, comprising:
an image acquisition processor configured to obtain an eye image under illumination of a light source;
an internal point determination processor, configured to obtain a first internal point in a pupil of the eye image;
a calculation processor, configured to calculate gradient changes of pixel points along a straight line starting from the first internal point toward outside of the pupil;
a first edge point determination processor, configured to determine a pixel point having a gradient change that satisfies a preset condition as a first edge point at an edge of the pupil;
a search processor, configured to search for a plurality of pixel points in the eye image having the same or similar gradient change as the gradient change at the first edge point, and set the plurality of pixel points as a second edge point to an Nth edge point, where N is a positive integer greater than 4; and
a fitting processor, configured to perform ellipse fitting on the edge points from the first edge point to the Nth edge point to obtain a pupil center.

### Claim 15 (depends on 14)

The pupil positioning apparatus according to claim 14, wherein the internal point determination unit comprises:
a centroid determination subprocessor, configured to obtain imaging points of the light source on a cornea and calculate an average centroid position of all the imaging points in the eye image;
a scan subprocessor, configured to obtain a pupil scan area based on the average centroid position and scan the eye image in the pupil scan area using a pupil scan frame having a predetermined side length based on a predetermined scan step size; and
an internal point determination subprocessor, configured to determine a center of the pupil scan frame having a smallest average gray value as the first internal point.

### Claim 16 (depends on 14)

The pupil positioning device according to claim 14, wherein the fitting processor comprises:
a grouping subprocessor, configured to divide the edge points from the first edge point to the Nth edge point into consecutive groups;
an ellipse fitting subprocessor, configured to perform ellipse fitting on the edge points in each of the groups separately to obtain a plurality of ellipses;
a pupil positioning subprocessor, configured to obtain intersection points of the plurality of ellipses and obtain the pupil center using the intersection points.

### Claim 17 (depends on 14)

A virtual reality/augmented reality apparatus, comprising the pupil positioning apparatus according to claim 14.

### Claim 18 (depends on 1)

A computer storage medium having stored thereon computer program instructions, wherein the pupil positioning method of claim 1 is implemented when the computer program instructions are executed by a processor.

### Claim 19

A pupil positioning apparatus, comprising:
a display screen;
a processor; and
a storage medium coupled to the processor and storing a computer program that, when executed by the processor, causes the processor to:
obtain an eye image under illumination of a light source;
determine a first internal point in a pupil of the eye image;
calculate gradient changes of pixel points along a straight line starting from the first internal point toward outside of the pupil;
determine a plurality of edge points at an edge of the pupil based on the gradient changes of the pixel points along the straight line; and
perform ellipse fitting on the edge points to obtain a pupil center.

### Claim 20 (depends on 19)

The pupil positioning apparatus according to claim 19, wherein the computer program, when executed by the processor, causes the processor to determine the first internal point in the pupil of the eye image comprises causing the processor to:
obtain imaging points of the light source on a cornea and calculate an average centroid position of all the imaging points in the eye image;
obtain a pupil scan area based on the average centroid position;
scan the eye image in the pupil scan area using a pupil scan frame having a predetermined side length according to a predetermined scan step size; and
determine a center of the pupil scan frame having a smallest average gray value as the first internal point.
