# US 12,008,159 B2 · Systems and methods for gaze-tracking > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,159 B2 - Document ID: US12008159B2 - Status: Granted - Kind: B2 - Application number: 18/113,222 - Filed: 2023-02-23 - Published: 2023-06-22 - Granted: 2024-06-11 - Earliest claimed: 2021-04-19 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Jatin Sharma; Jonathan T. Campbell; Jay C. Beavers; Peter John Ansell - Web page: https://patentgrep.com/patent/US12008159B2 ## CPC - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06V 40/18: Eye characteristics, e.g. of the iris - G06F 3/012: {Head tracking input arrangements} - G06F 18/24133: {Distances to prototypes} - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06T 7/73: using feature-based methods - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06T 2207/10024: Color image - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30201: Face - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Systems and methods are provided for predicting an eye gaze location of an operator of a computing device. In particular, the method generates an image grid that includes regions of interest based on a facial image. The facial image is based on a received image frame of a video stream that captures the operator using the computing device. The image grid further includes a region that indicate rotation information of the face. The method further uses a combination of trained neural networks to extract features of the regions of interest in the image grid and predict the eye gaze location on the screen of the computing device. The trained set of neural networks includes a convolutional neural network. The method optionally generate head pose pitch, roll, and yaw information to improve accuracy of predicting the location of an eye gaze. ## Claims ### Claim 1 A computer-implemented method for predicting a location of an eye gaze of an operator, the method comprising: receiving an input image, wherein the input image depicting the operator gazing at a location on a screen of a device; extracting a plurality of regions from the input image, wherein the plurality of regions include: one or more eyes, and a region indicating at least a position and a rotation of a face; generating, based on the extracted plurality of regions, an image grid, wherein the image grid includes a plurality of image data, the plurality of image data including image data that correspond to distinct regions of the plurality of regions; determining, based on a previously trained gaze-data prediction model and using the image grid, a location of the eye gaze, wherein the location includes a multi-dimensional location of the eye gaze associated with the screen of the device; and causing, based on the location, one or more actions associated with an operation of the device. ### Claim 2 (depends on 1) The computer-implemented method according to claim 1, the method further comprising: generating, based on the input image, head pose information of the operator, wherein the head pose information includes a direction of a head of the operator relative to the screen of the device; generating, based on the image grid and the previously trained gaze-data prediction model, a multi-dimensional vector, wherein the previously trained gaze-data prediction model includes a combination of a convolutional neural network and a fully-connected network; and generating, based on a combination of the multi-dimensional vector and the head pose information using the fully-connected network, the location of the eye gaze. ### Claim 3 (depends on 1) The computer-implemented method according to claim 1, the method further comprising: transmitting the location of the eye gaze to an output controller, causing an update to a gaze location indicator on the screen of the device. ### Claim 4 (depends on 1) The computer-implemented method according to claim 1, the method further comprising: receiving the input image from an RGB camera, wherein the input image is based on an RGB color space. ### Claim 5 (depends on 1) The computer-implemented method according to claim 1, wherein the image grid includes a combination of: the one or more eyes, the face, a nose, one or more ears, and the region indicating at least the position and the rotation of the face. ### Claim 6 (depends on 2) The computer-implemented method according to claim 2, wherein the head pose information includes one or more of a two-dimensional coordinates and a three-dimensional locations of one or more of: a corner of an eye, a tip of a nose, a corner of a mouth, or a tip of a chin. ### Claim 7 (depends on 2) The computer-implemented method according to claim 2, wherein the convolutional neural network predicts one or more features of the one or more eyes based on one or more of: a feature of the face, and a feature of the region indicating at least the position and the rotation of the face in the image grid. ### Claim 8 A system for predicting a location of an eye gaze of an operator, the system comprising: a processor; and a memory storing computer-executable instructions that when executed by the processor cause the system to: receive an input image, wherein the input image depicting the operator gazing at a location on a screen of a device; extract a plurality of regions from the input image, wherein the plurality of regions include: one or more eyes of the operator, and a region indicating at least a position and a rotation of a face of the operator; generate, based on the extracted plurality of regions, an image grid, wherein the image grid includes a plurality of image data, the plurality of image data including image data that correspond to distinct regions of the plurality of regions; determine, based on a previously trained gaze-data prediction model and using the image grid, a location of the eye gaze, wherein the location includes a multi-dimensional location of the eye gaze associated with the screen of the device; and cause, based on the location, one or more actions associated with an operation of the device. ### Claim 9 (depends on 8) The system according to claim 8, the computer-executable instructions that when executed by the processor further cause the system to: generate, based on the input image, head pose information of the operator, wherein the head pose information includes a direction of a head of the operator relative to the screen of the device; generate, based on the image grid and the previously trained gaze-data prediction model, a multi-dimensional vector, wherein the previously trained gaze-data prediction model includes a combination of a convolutional neural network and a fully-connected network; and generate, based on a combination of the multi-dimensional vector and the head pose information using the fully-connected network, the location of the eye gaze. ### Claim 10 (depends on 8) The system according to claim 8, the computer-executable instructions that when executed by the processor further cause the system to: transmit the location of the eye gaze to an output controller, causing an update to a gaze location indicator on the screen of the device. ### Claim 11 (depends on 8) The system according to claim 8, the computer-executable instructions that when executed by the processor further cause the system to: receive the input image from an RGB camera, wherein the input image is based on an RGB color space. ### Claim 12 (depends on 8) The system according to claim 8, wherein the image grid includes a combination of: the one or more eyes, the face, a nose, one or more ears, and the region indicating at least the position and the rotation of the face. ### Claim 13 (depends on 9) The system according to claim 9, wherein the head pose information includes one or more of a two-dimensional coordinates and a three-dimensional locations of one or more of: a corner of an eye, a tip of a nose, a corner of a mouth, or a tip of a chin. ### Claim 14 (depends on 9) The system according to claim 9, wherein the convolutional neural network predicts one or more features of the one or more eyes based on one or more of: a feature of the face, and a feature of the region indicating at least the position and the rotation of the face in the image grid. ### Claim 15 A computer-readable storage medium storing computer-executable instructions that when executed by a processor cause a computer system to: receive an input image, wherein the input image depicting an operator gazing at a location on a screen of a device; extract a plurality of regions from the input image, wherein the plurality of regions include: one or more eyes, and a region indicating at least a position and a rotation of a face; generate, based on the extracted plurality of regions, an image grid, wherein the image grid includes a plurality of image data, the plurality of image data including image data that correspond to distinct regions of the plurality of regions; determine, based on a previously trained gaze-data prediction model and using the image grid, a location of an eye gaze, wherein the location includes a multi-dimensional location of the eye gaze associated with the screen of the device; and cause, based on the location, one or more actions associated with an operation of the device. ### Claim 16 (depends on 15) The computer-readable storage medium of claim 15, the computer-executable instructions that when executed by the processor further cause the computer system to: generate, based on the input image, head pose information of the operator, wherein the head pose information includes a direction of a head of the operator relative to the screen of the device; generate, based on the image grid and the previously trained gaze-data prediction model, a multi-dimensional vector, wherein the previously trained gaze-data prediction model includes a combination of a convolutional neural network and a fully-connected network; and generate, based on a combination of the multi-dimensional vector and the head pose information using the fully-connected network, the location of the eye gaze. ### Claim 17 (depends on 15) The computer-readable storage medium of claim 15, the computer-executable instructions that when executed by the processor further cause the computer system to: transmit the location of the eye gaze to an output controller, causing an update to a gaze location indicator on the screen of the device. ### Claim 18 (depends on 15) The computer-readable storage medium of claim 15, the computer-executable instructions that when executed by the processor further cause the computer system to: receive the input image from an RGB camera, wherein the input image is based on an RGB color space. ### Claim 19 (depends on 16) The computer-readable storage medium of claim 16, wherein the head pose information includes one or more of a two-dimensional coordinates and a three-dimensional locations of one or more of: a corner of an eye, a tip of a nose, a corner of a mouth, or a tip of a chin. ### Claim 20 (depends on 16) The computer-readable storage medium of claim 16, wherein the image grid includes a combination of: the one or more eyes, the face, a nose, one or more ears, and the region indicating at least the position and the rotation of the face, and wherein the convolutional neural network predicts one or more features of the one or more eyes based on one or more of: a feature of the face, and a feature of the region indicating at least the position and the rotation of the face in the image grid.