# US 11,998,335 B2 · Systems and methods of capturing eye-gaze data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,998,335 B2
- Document ID: US11998335B2
- Status: Granted
- Kind: B2
- Application number: 17/234,568
- Filed: 2021-04-19
- Published: 2022-10-20
- Granted: 2024-06-04
- Earliest claimed: 2021-04-19
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Jatin Sharma; Jonathan T. Campbell; Jay C. Beavers; Peter John Ansell
- Web page: https://patentgrep.com/patent/US11998335B2

## CPC

- A61B 5/163: {by tracking eye movement, gaze, or pupil change}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- A61B 5/7267: {involving training the classification device}
- A61B 5/7435: {Displaying user selection data, e.g. icons in a graphical user interface}
- G06F 3/1423: {controlling a plurality of local displays, e.g. CRT and flat panel display}
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems and methods are provided for collecting eye-gaze data for training an eye-gaze prediction model. The collecting includes selecting a scan path that passes through a series of regions of a grid on a screen of a computing device, moving a symbol as an eye-gaze target along the scan path, and receiving facial images at eye-gaze points. The eye-gaze points are uniformly distributed within the respective regions. Areas of the regions that are adjacent to edges and corners of the screen are smaller than other regions. The difference in areas shifts centers of the regions toward the edges, density of data closer to the edges. The scan path passes through locations in proximity to the edges and corners of the screen for capturing more eye-gaze points in the proximity. The methods interactively enhance variations of facial images by displaying instructions to the user to make specific actions associated with the face.

## Claims

### Claim 1

A computer-implemented method for collecting eye-gaze data as training data for an eye-gaze prediction model, the method comprising:
selecting a scan path from a set of predetermined scan paths, wherein each scan path is non-self-overlapping on a screen of a device, the scan path passes through a point in a region of a series of regions in a grid on the screen, and the point represents an expected value of uniformly distributed random eye-gaze points in the region;
displaying a symbol as an eye-gaze target on the screen, wherein the eye-gaze target moves along the scan path for guiding attention of the operator;
receiving a combination of eye-gaze point data and input images associated with a plurality of points along the scan path as training data for the eye-gaze prediction model;
training the eye-gaze prediction model using the training data, wherein the eye-gaze prediction model includes data associated with parameters in one or more neural networks; and
updating the parameters in the one or more neural networks using the trained eye-gaze prediction model.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein a horizontal-vertical dimension ratio of each of the series of regions in the grid and a horizontal-vertical dimensional ratio of the screen are identical.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein areas of one or more regions in the series of regions are identical.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein areas of one or more regions in the series of regions adjacent to at least an edge of the screen are smaller than other regions in the series of the regions.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the screen includes a plurality of screens, and wherein the scan path traverses across a series of regions in a grid on the plurality of screens.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, the method further comprising:
displaying the symbol without movement, wherein the symbol is on a scan path;
selecting an icon from a set of icons, wherein the set of icons include an unrestricted movement of a face, a restricted movement of the face, and one or more actions, wherein the one or more actions include change distance, change position, change rotation, and change sides of a face of the operator;
displaying the icon at a location adjacent to the symbol; and
interactively receiving, in response to the displaying the icon, one or more input images of the operator.

### Claim 7

A system for collecting eye-gaze data as training data for an eye-gaze prediction model, the system comprising:
a processor; and
a memory storing computer-executable instructions that when executed by the processor cause the system to:
select a scan path from a set of predetermined scan paths, wherein each scan path is non-self-overlapping on a screen of a device, the scan path passes through a point in a region of a series of regions in a grid on the screen, and the point represents an expected value of uniformly distributed random eye-gaze pints in the region;
display a symbol as an eye-gaze target on the screen, wherein the eye-gaze target moves along the scan path for guiding attention of the operator;
receive a combination of eye-gaze data and input images associated with a plurality of points along the scan path as training data for the eye-gaze prediction model;
train the eye-gaze prediction model using the training data, wherein the eye-gaze prediction model includes data associated with parameters in one or more neural networks; and
update the parameters in the one or more neural networks using the trained eye-gaze prediction model.

### Claim 8 (depends on 7)

The system according to claim 7, wherein a horizontal-vertical dimension ratio of each of the series of regions in the grid and a horizontal-vertical dimensional ratio of the screen are identical.

### Claim 9 (depends on 7)

The system according to claim 7, wherein areas of one or more regions in the series of regions are identical.

### Claim 10 (depends on 7)

The system according to claim 7, wherein areas of one or more regions in the series of regions adjacent to at least an edge of the screen are smaller than other regions in the series of the regions.

### Claim 11 (depends on 7)

The system according to claim 7, wherein the screen includes a plurality of screens, and wherein the scan path traverses across a series of regions in a grid on the plurality of screens.

### Claim 12 (depends on 7)

The system according to claim 7, the computer-executable instructions that when executed by the processor further cause the system to:
display the symbol without movement, wherein the symbol is on a scan path;
select an icon from a set of icons, wherein the set of icons include an unrestricted movement of a face, a restricted movement of the face, and one or more actions, wherein the one or more actions include change distance, change position, change rotation, and change sides of a face of the operator;
display the icon at a location adjacent to the symbol; and
interactively receive, in response to the displaying the icon, one or more input images of the operator.

### Claim 13

A computer-readable recording medium storing computer-executable instructions that when executed by a processor cause a computer system to:
select a scan path from a set of predetermined scan paths, wherein each scan path is non-self-overlapping on a screen of a device, the scan path passes through a point in a region of a series of regions in a grid on the screen, and the point represents an expected value of uniformly distributed random eye-gaze points in the region;
display a symbol as an eye-gaze target on the screen, wherein the eye-gaze target moves along the scan path for guiding attention of the operator;
receive a combination of eye-gaze data and input images associated with a plurality of points along the scan path as training data for the eye-gaze prediction model;
train the eye-gaze prediction model using the training data, wherein the eye-gaze prediction model includes data associated with parameters in one or more neural networks; and
update the parameters in the one or more neural networks using the trained eye-gaze prediction model.

### Claim 14 (depends on 13)

The computer-readable non-transitory recording medium of claim 13, wherein a horizontal-vertical dimension ratio of each of the series of regions in the grid and a horizontal-vertical dimensional ratio of the screen are identical.

### Claim 15 (depends on 13)

The computer-readable non-transitory recording medium of claim 13, wherein areas of one or more regions in the series of regions adjacent to at least an edge of the screen are smaller than other regions in the series of the regions.

### Claim 16 (depends on 13)

The computer-readable non-transitory recording medium of claim 13, wherein the screen includes a plurality of screens, and wherein the scan path traverses across a series of regions in a grid on the plurality of screens.

### Claim 17 (depends on 13)

The computer-readable non-transitory recording medium of claim 13, the computer-executable instructions that when executed by the processor further cause the system to:
display the symbol without movement, wherein the symbol is on a scan path;
select an icon from a set of icons, wherein the set of icons include an unrestricted movement of a face, a restricted movement of the face, and one or more actions, wherein the one or more actions include change distance, change position, change rotation, and change sides of a face of the operator;
display the icon at a location adjacent to the symbol; and
interactively receive, in response to the displaying the icon, one or more input images of the operator.