# US 2021/0333872 A1 · Multimodal eye tracking

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0333872 A1
- Document ID: US20210333872A1
- Status: Published application
- Kind: A1
- Application number: 17/239,451
- Filed: 2021-04-23
- Published: 2021-10-28
- Earliest claimed: 2017-04-14
- Assignees: Magic Leap, Inc.
- Applicants: Magic Leap, Inc.
- Inventors: Edwin Joseph SELKER; Ivan YEOH
- Web page: https://patentgrep.com/patent/US20210333872A1

## CPC

- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G02B 27/017: {Head mounted}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method is disclosed, the method comprising the steps of receiving, at a first time interval from a first sensor configured to output data indicative of a first position of an eye, first data; receiving, at a second time interval from a second sensor configured to output data indicative of a delta position of the eye, second data; determining, based on the first data, a first position of the eye; determining, based on the second data, a delta position of the eye; determining, using the first position of the eye and the delta position of the eye, a second absolute position of the eye; and in response to determining the second position of the eye, generating an output signal indicative of the second position of the eye.

## Claims

### Claim 1

A method comprising:
receiving first data indicative of a first position of an eye, the first data received at a first time interval, the first data output by a first sensor that is configured to output data indicative of the first position of the eye;
receiving second data indicative of a second position of the eye, the second data received at a second time interval, the second data output by a second sensor that is configured to output data indicative of the second position of the eye, wherein the second sensor is distinct from the first sensor, wherein the first sensor is configured to obtain the first data at a first frequency, and wherein the second sensor is configured to obtain the second data at a second frequency that is higher than the first frequency;
determining, based on the first data, the first position of the eye;
determining, based on the second data, the second position of the eye;
determining, based on the first position of the eye and the second position of the eye, a third position of the eye; and
generating an output signal indicative of the third position of the eye.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first sensor comprises an optical sensor.

### Claim 3 (depends on 1)

The method of claim 1, wherein the second sensor comprises an electrooculography sensor.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first sensor operates in a low-power mode during the first time interval.

### Claim 5 (depends on 1)

The method of claim 1, wherein the second sensor operates in a low-power mode during the second time interval.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
determining, based on the third position of the eye, a first eye movement behavior.

### Claim 7 (depends on 6)

The method of claim 6, wherein the first eye movement behavior comprises saccadic movement, smooth pursuit, fixation, nystagmus, or vestibulo-ocular movement.

### Claim 8 (depends on 6)

The method of claim 6, further comprising:
determining, based on the first eye movement behavior, a third time interval at which to receive data from the first sensor, and
determining, based on the first eye movement behavior, a fourth time interval at which to receive data from the second sensor.

### Claim 9 (depends on 6)

The method of claim 6, wherein determining the first eye movement behavior comprises:
generating a confidence score corresponding to a likelihood of the first eye movement behavior;
comparing the confidence score to a threshold value; and
determining that the confidence score exceeds the threshold value.

### Claim 10 (depends on 6)

The method of claim 6, further comprising receiving, from a third sensor, third data and wherein the first eye movement behavior is determined based on the third data.

### Claim 11 (depends on 10)

The method of claim 10, wherein the third sensor comprises an accelerometer, a gyroscope, an electronic compass, a magnetometer, or an inertial measurement unit.

### Claim 12 (depends on 10)

The method of claim 10, wherein the third sensor comprises a GPS sensor.

### Claim 13 (depends on 10)

The method of claim 10, wherein the third sensor comprises an ambient light sensor.

### Claim 14 (depends on 10)

The method of claim 10, wherein the first eye movement behavior is determined using a neural network.

### Claim 15 (depends on 10)

The method of claim 10, further comprising training a neural network using information comprising the first data, the second data, the third data, the third position of the eye, or the first eye movement behavior.

### Claim 16 (depends on 15)

The method of claim 15, further comprising determining a second eye movement behavior based on the neural network.

### Claim 17 (depends on 1)

The method of claim 1, wherein the first sensor and the second sensor are attached to a head-mounted device comprising a display.

### Claim 18 (depends on 17)

The method of claim 17, further comprising in response to determining the third position of the eye:
determining a region of the display corresponding to the second position of the eye, the region having a display state equal to a first display state; and
changing the display state of the region from the first display state to a second display state.

### Claim 19 (depends on 1)

The method of claim 1, further comprising determining a delta status that the delta position of the eye is less than a threshold,
wherein the first sensor is configured to operate in a low-power mode in response to the determined delta status.

### Claim 20

A system comprising:
a first sensor configured for outputting first data indicative of a first position of an eye, wherein the first sensor is configured to obtain the first data at a first frequency;
a second sensor configured for outputting second data indicative of a second position of the eye, wherein the second sensor is distinct from the first sensor, and wherein the second sensor is configured to obtain the second data at a second frequency that is higher than the first frequency;
one or more processors operatively coupled to the first sensor and the second sensor, the one or more processors configured to perform a method comprising:
receiving the first data indicative of the first position of the eye, the first data to be received at a first time interval;
receiving the second data indicative of the second position of the eye, the second data to be received at a second time interval,
determining, based on the first data, the first position of the eye;
determining, based on the second data, the second position of the eye;
determining, based on the first position of the eye and the second position of the eye, a third position of the eye; and
generating an output signal indicative of the third position of the eye.

### Claim 21 (depends on 20)

The system of claim 20, wherein the first sensor comprises an optical sensor.

### Claim 22 (depends on 20)

The system of claim 20, wherein the second sensor comprises an electrooculography sensor.

### Claim 23 (depends on 20)

The system of claim 20, wherein the first sensor is configured to operate in a low-power mode during the first time interval.

### Claim 24 (depends on 20)

The system of claim 20, wherein the second sensor is configured to operate in a low-power mode during the second time interval.

### Claim 25 (depends on 20)

The system of claim 20, wherein the method further comprises:
determining, based on the third position of the eye, a first eye movement behavior.

### Claim 26 (depends on 25)

The system of claim 25, wherein the method further comprises:
determining, based on the first eye movement behavior, a third time interval at which to receive data from the first sensor, and
determining, based on the first eye movement behavior, a fourth time interval at which to receive data from the second sensor.

### Claim 27 (depends on 25)

The system of claim 25, further comprising a third sensor, wherein the method further comprises receiving, from the third sensor, third data, and wherein the first eye movement behavior is determined based on the third data.

### Claim 28 (depends on 27)

The system of claim 27, wherein the third sensor comprises an accelerometer, a gyroscope, an electronic compass, a magnetometer, an inertial measurement unit, a GPS sensor, or an ambient light sensor.

### Claim 29 (depends on 20)

The system of claim 20, wherein the first sensor and the second sensor are attached to a head-mounted device comprising a display.

### Claim 30 (depends on 29)

The system of claim 29, wherein the method further comprises:
determining a region of the display corresponding to the third position of the eye, the region having a display state equal to a first display state; and
changing the display state of the region from the first display state to a second display state.

### Claim 31 (depends on 20)

The system of claim 20, wherein the method further comprises determining a delta status based on the first position of the eye and on the second position of the eye, and wherein the first sensor is configured to operate in a low-power mode in response to the determined delta status.

### Claim 32

A non-transitory computer-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising:
receiving first data indicative of a first position of an eye, the first data received at a first time interval, the first data output by a first sensor that is configured to output data indicative of the first position of the eye;
receiving second data indicative of a second position of the eye, the second data received at a second time interval, the second data output by a second sensor that is configured to output data indicative of the second position of the eye, wherein the second sensor is distinct from the first sensor, wherein the first sensor is configured to obtain the first data at a first frequency, and wherein the second sensor is configured to obtain the second data at a second frequency that is higher than the first frequency;
determining, based on the first data, the first position of the eye;
determining, based on the second data, the second position of the eye;
determining, based on the first position of the eye and the second position of the eye, a third position of the eye; and
generating an output signal indicative of the third position of the eye.

### Claim 33 (depends on 32)

The non-transitory computer-readable medium of claim 32, wherein the first sensor comprises an optical sensor.

### Claim 34 (depends on 32)

The non-transitory computer-readable medium of claim 32, wherein the second sensor comprises an electrooculography sensor.

### Claim 35 (depends on 32)

The non-transitory computer-readable medium of claim 32, wherein the first sensor is configured to operate in a low-power mode during the first time interval.

### Claim 36 (depends on 32)

The non-transitory computer-readable medium of claim 32, wherein the second sensor is configured to operate in a low-power mode during the second time interval.

### Claim 37 (depends on 32)

The non-transitory computer-readable medium of claim 32, the method further comprising:
determining, based on the second position of the eye, a first eye movement behavior.

### Claim 38 (depends on 37)

The non-transitory computer-readable medium of claim 37, the method further comprising:
determining, based on the first eye movement behavior, a third time interval at which to receive data from the first sensor, and
determining, based on the first eye movement behavior, a fourth time interval at which to receive data from the second sensor.

### Claim 39 (depends on 37)

The non-transitory computer-readable medium of claim 37, wherein the method further comprises receiving, from a third sensor, third data, and wherein the first eye movement behavior is determined based on the third data.

### Claim 40 (depends on 39)

The non-transitory computer-readable medium of claim 39, wherein the third sensor comprises an accelerometer, a gyroscope, an electronic compass, a magnetometer, an inertial measurement unit, a GPS sensor, or an ambient light sensor.

### Claim 41 (depends on 32)

The non-transitory computer-readable medium of claim 32, wherein the first sensor and the second sensor are attached to a head-mounted device comprising a display.

### Claim 42 (depends on 41)

The non-transitory computer-readable medium of claim 41, the method further comprising:
determining a region of the display corresponding to the third position of the eye, the region having a display state equal to a first display state; and
changing the display state of the region from the first display state to a second display state.

### Claim 43 (depends on 32)

The non-transitory computer-readable medium of claim 32, wherein the method further comprises determining a delta status based on the first position of the eye and on the second position of the eye, and
wherein the first sensor is configured to operate in a low-power mode in response to the determined delta status.
