# US 2016/0259961 A1 · Pupil detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0259961 A1
- Document ID: US20160259961A1
- Status: Published application
- Kind: A1
- Application number: 15/157,675
- Filed: 2016-05-18
- Published: 2016-09-08
- Earliest claimed: 2014-07-24
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Milan Vugdelija; Djordje Nijemcevic; David Zachris Nister; Christopher Maurice Mei; Chandra Sekhar Gatla
- Web page: https://patentgrep.com/patent/US20160259961A1

## CPC

- G06V 40/19: Sensors therefor
- G06F 18/23: Clustering techniques
- G06T 7/73: using feature-based methods
- G06T 7/90: Determination of colour characteristics
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 40/193: {Preprocessing; Feature extraction}
- G06T 2207/10024: Color image
- G06T 2207/30201: Face
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Embodiments that relate to determining an estimated pupil region of an eye are disclosed. In one embodiment a method includes receiving an image of an eye, with the image comprising a plurality of pixels. A rough pupil region is generated using at least a subset of the plurality of pixels. A plurality of pupil boundary point candidates are extracted from the rough pupil region, with each of the candidates weighted based on color values of at least two neighbor pixels. A parametric curve may be fitted to the weighted pupil boundary point candidates to determine the estimated pupil region of the eye of the user.

## Claims

### Claim 1

A method for determining an estimated pupil region of an eye, the method comprising:
receiving an image of the eye comprising a plurality of pixels;
for each of a plurality of pupil boundary point candidate pixels extracted from a rough pupil region generated from a subset of the plurality of pixels, weighting the pupil boundary point candidate pixel based on color values of at least two neighbor pixels to generate weighted pupil boundary point candidates; and
fitting a parametric curve to the weighted pupil boundary point candidates to determine the estimated pupil region of the eye.

### Claim 2 (depends on 1)

The method of claim 1, wherein weighting each of the pupil boundary point candidate pixels based on color values of at least two neighbor pixels further comprises:
determining a color value of each of a plurality of inner pixels located between the pupil boundary point candidate pixel and an estimated center of the rough pupil region along a ray extending from the estimated center through the pupil boundary point candidate pixel; and
determining a color value of each of a plurality of outer pixels located along the ray, the plurality of outer pixels located on an opposite side of the pupil boundary point candidate pixel from the inner pixels.

### Claim 3 (depends on 2)

The method of claim 2, wherein weighting each of the pupil boundary point candidate pixels based on color values of at least two neighbor pixels further comprises averaging the color values of the plurality of inner pixels, and averaging the color values of the plurality of outer pixels.

### Claim 4 (depends on 2)

The method of claim 2, wherein weighting each of the pupil boundary point candidate pixels further comprises:
clustering the pupil boundary point candidate pixels in a two-dimensional color space according to an average color value of their inner pixels and an average color value of their outer pixels; and
assigning a weight to each of the pupil boundary point candidate pixels based on a distance of each of the pupil boundary point candidate pixels from a center of a dominant cluster in the two-dimensional color space.

### Claim 5 (depends on 1)

The method of claim 1, wherein fitting the parametric curve to the weighted pupil boundary point candidates further comprises:
minimizing a cost function that is computed using each of the weighted pupil boundary point candidates.

### Claim 6 (depends on 5)

The method of claim 5, wherein one or more elements contributing to the cost function comprise one or more of a discrepancy between an image-space gradient and a parametric curve normal for each of the weighted pupil boundary point candidates, and a distance of each of the weighted pupil boundary point candidates from the parametric curve.

### Claim 7 (depends on 1)

The method of claim 1, wherein generating the rough pupil region further comprises extracting one or more stable regions from the image.

### Claim 8 (depends on 7)

The method of claim 7, wherein extracting one or more stable regions from the image further comprises identifying one or more maximally stable extremal regions.

### Claim 9 (depends on 1)

The method of claim 1, wherein generating the rough pupil region further comprises using one or more classifiers to classify each of the pixels of the subset of the plurality of pixels into one of a plurality of classification regions.

### Claim 10 (depends on 1)

The method of claim 1, wherein the rough pupil region comprises one or more connected components, and the method further comprises scoring each of the connected components using a plurality of heuristics corresponding to different pupil characteristics.

### Claim 11

A system for determining an estimated pupil region of an eye, the system comprising:
a computing device; and
a pupil detection module executed by a processor of the computing device, the pupil detection module configured to:
receive an image of the eye comprising a plurality of pixels;
for each of a plurality of pupil boundary point candidate pixels extracted from a rough pupil region generated from a subset of the plurality of pixels, weight the pupil boundary point candidate pixel based on color values of at least two neighbor pixels to generate weighted pupil boundary point candidates; and
fit a parametric curve to the weighted pupil boundary point candidates to determine the estimated pupil region of the eye.

### Claim 12 (depends on 11)

The system of claim 11, wherein to weight each of the pupil boundary point candidate pixels based on the color values of at least two neighbor pixels, the pupil detection module is further configured to:
determine a color value of each of a plurality of inner pixels located between the pupil boundary point candidate pixel and an estimated center of the rough pupil region along a ray extending from the estimated center through the pupil boundary point candidate pixel; and
determine a color value of each of a plurality of outer pixels located along the ray, the plurality of outer pixels located on an opposite side of the pupil boundary point candidate pixel from the inner pixels.

### Claim 13 (depends on 12)

The system of claim 12, wherein to weight each of the pupil boundary point candidate pixels based on the color values of at least two neighbor pixels, the pupil detection module is further configured to:
average the color values of the plurality of inner pixels, and average the color values of the plurality of outer pixels.

### Claim 14 (depends on 12)

The system of claim 12, wherein to weight each of the pupil boundary point candidate pixels based on the color values of at least two neighbor pixels, the pupil detection module is further configured to:
cluster the pupil boundary point candidate pixels in a two-dimensional color space according to their inner pixel values and their outer pixels values; and
assign a weight to each of the pupil boundary point candidate pixels based on a distance of each of the pupil boundary point candidate pixels from a center of a dominant cluster in the two-dimensional color space.

### Claim 15 (depends on 11)

The system of claim 11, wherein to fit the parametric curve to the weighted pupil boundary point candidates, the pupil detection module is further configured to minimize a cost function that is computed using each of the weighted pupil boundary point candidates.

### Claim 16 (depends on 15)

The system of claim 15, wherein one or more elements contributing to the cost function comprise one or more of a discrepancy between an image-space gradient and a parametric curve normal for each of the weighted pupil boundary point candidates, and a distance of each of the weighted pupil boundary point candidates from the parametric curve.

### Claim 17 (depends on 11)

The system of claim 11, wherein to generate the rough pupil region the pupil detection module is further configured to extract one or more stable regions from the image.

### Claim 18 (depends on 17)

The system of claim 17, wherein to extract one or more stable regions from the image the pupil detection module is further configured to identify one or more maximally stable extremal regions.

### Claim 19 (depends on 11)

The system of claim 11, wherein to generate the rough pupil region the pupil detection module is further configured to use one or more classifiers to classify each of the pixels of the subset of the plurality of pixels into one of a plurality of classification regions.

### Claim 20

A method for determining an estimated pupil region of an eye, the method comprising:
receiving an image of the eye comprising a plurality of pixels;
weighting each of a plurality of pupil boundary point candidate pixels extracted from a rough pupil region generated from a subset of the plurality of pixels to generate weighted pupil boundary point candidates, wherein weighting each of the pupil boundary point candidate pixels comprises:
determining a color value of each of a plurality of inner pixels located between the pupil boundary point candidate pixel and an estimated center of the rough pupil region along a ray extending from the estimated center through the pupil boundary point candidate pixel; and
determining a color value of each of a plurality of outer pixels located along the ray, the plurality of outer pixels located on an opposite side of the pupil boundary point candidate pixel from the inner pixels; and
fitting a parametric curve to the weighted pupil boundary point candidates to determine the estimated pupil region of the eye.
