# US 11,907,418 B2 · Eye tracking in near-eye displays

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,907,418 B2
- Document ID: US11907418B2
- Status: Granted
- Kind: B2
- Application number: 17/345,305
- Filed: 2021-06-11
- Published: 2021-12-23
- Granted: 2024-02-20
- Earliest claimed: 2020-06-12
- Assignees: Magic Leap, Inc.
- Applicants: University of Washington; Magic Leap, Inc.
- Inventors: Eric J. Seibel; Steven L. Brunton; Chen Gong; Brian T. Schowengerdt
- Web page: https://patentgrep.com/patent/US11907418B2

## CPC

- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 27/0172: {characterised by optical features}
- G02B 27/0179: {Display position adjusting means not related to the information to be displayed}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/00: Machine learning
- G06T 7/248: {involving reference images or patches}
- G06T 7/277: involving stochastic approaches, e.g. using Kalman filters
- G06T 7/33: using feature-based methods
- G06T 7/74: {involving reference images or patches}
- G06V 10/764: using classification, e.g. of video objects
- G06V 40/18: Eye characteristics, e.g. of the iris
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30201: Face
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Techniques for tracking eye movement in an augmented reality system identify a plurality of base images of an object or a portion thereof. A search image may be generated based at least in part upon at least some of the plurality of base images. A deep learning result may be generated at least by performing a deep learning process on a base image using a neural network in a deep learning mode. A captured image may be localized at least by performing an image registration process on the captured image and the search image using a Kalman filter model and the deep learning result.

## Claims

### Claim 1

A method for tracking eye movement in a near-eye display, comprising:
identifying a plurality of base images of an object or a portion thereof;
generating a search image based at least in part upon at least some of the plurality of base images;
generating a deep learning result at least by performing a deep learning process on a base image using a neural network in a deep learning model; and
localizing a captured image to the search image at least by performing an image registration process on the captured image and the search image using a Kalman filter model and the deep learning result,
wherein the object comprises a retina of a user, and the captured image representing at least a portion of the retina is localized with respect to the search image without using pupil-glint techniques that capture glint from the pupil in response to one or more input light patterns.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating the deep learning result comprises:
extracting a feature in a region in the captured image using a deep convolution neural network (DCNN);
converting the feature into a plurality of features that comprises a first feature and a second feature;
classifying the region into a positive region or a negative region based at least in part upon the first feature; and
determining regression or correction for the region based at least in part upon the second feature.

### Claim 3 (depends on 1)

The method of claim 1, wherein generating the deep learning result comprises:
extracting a feature in a region in the captured image using a deep convolution neural network (DCNN); and
converting the feature into a plurality of features that comprises a first feature and a second feature.

### Claim 4 (depends on 3)

The method of claim 3, wherein generating the deep learning result further comprises:
generating a first response map at least by convolving at least the first feature using one or more convolution layers in the DCNN; and
classifying the region into a positive region or a negative region based at least in part upon the first response map.

### Claim 5 (depends on 4)

The method of claim 4, wherein generating the deep learning result further comprises:
generating a second response map at least by convolving the second feature using the one or more convolution layers or one or more different convolution layers in the DCNN; and
determining regression or correction for a predicted location of the region using at least the second response map.

### Claim 6

A method for tracking eye movement in a near-eye display, comprising:
identifying a plurality of base images of an object or a portion thereof;
generating a search image based at least in part upon at least some of the plurality of base images;
generating a deep learning result at least by performing a deep learning process on a base image using a neural network in a deep learning model; and
localizing a captured image to the search image at least by performing an image registration process on the captured image and the search image using a Kalman filter model and the deep learning result,
wherein localizing the captured image comprises:
embedding the deep learning process into a state transition model; and
receiving the deep learning result at the Kalman filter model as one or more measurements using at least the state transition model, where the state transition model is used to determine a next position state at a next time point based at least in part upon a prior position state at a prior time point and at least one of process noise or a control vector of a state estimation model.

### Claim 7 (depends on 6)

The method of claim 6, wherein localizing the captured image comprises:
determining a measurement at a next time point by using at least the Kalman filter model in the image registration process;
reducing interference of similar background or one or more similar features at least by performing the image registration process; and
localizing the captured image in the search image using the Kalman filter model and the image registration process.

### Claim 8 (depends on 6)

The method of claim 6, wherein embedding the deep learning process comprises:
determining one or more control vectors for the state transition model;
deriving process noise from a statistical distribution; and
determining a time step for the state transition model.

### Claim 9 (depends on 8)

The method of claim 8, wherein localizing the captured image in the search image using the Kalman filter model and the image registration process comprises:
performing a coarse registration for an entire frame of the captured image; and
performing a fine registration for one or more features or feature points in an outer region around a region referenced in the coarse registration.

### Claim 10 (depends on 9)

The method of claim 9, wherein performing the coarse registration for the entire frame of the captured image comprises:
detecting the one or more features or the feature points in one or more regions in the entire frame of the captured image or the search image; and
registering the captured image to the one or more regions in the entire frame.

### Claim 11 (depends on 10)

The method of claim 10, wherein performing the fine registration for the one or more features or feature points in the outer region around a region referenced in the coarse registration comprises:
selecting a feature or feature point within an enlarged region that corresponding to the region that is referenced in the coarse registration; and
matching the feature or the feature point with a corresponding feature or feature point in the outer region of the search image.

### Claim 12 (depends on 11)

The method of claim 11, wherein performing the fine registration for the one or more features or feature points in the outer region around a region referenced in the coarse registration comprises:
localizing the captured image in the search image based at least in part upon a result of matching the feature or the feature point with a corresponding feature or feature point in the outer region of the search image.

### Claim 13 (depends on 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)

A system for tracking eye movement in a near-eye display, comprising a processor, a scanning fiber assembly, and memory storing thereupon instructions, which, when executed by the processor, causes the processor to perform any of the methods in claims 1 and 2-12 for tracking eye movement in a near-eye display.

### Claim 14 (depends on 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)

A non-transitory computer-readable medium storing thereupon instructions which, when executed by a microprocessor, causes the microprocessor to perform any of the methods in claims 1 and 2-12 for tracking eye movement in a near-eye display.