# US 12,217,421 B2 · Point of gaze tracking with integrated calibration process > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,421 B2 - Document ID: US12217421B2 - Status: Granted - Kind: B2 - Application number: 18/397,535 - Filed: 2023-12-27 - Published: 2024-07-11 - Granted: 2025-02-04 - Earliest claimed: 2023-01-05 - Assignees: NeuraLight Ltd. - Applicants: NeuraLight Ltd. - Inventors: Eran Harpaz - Web page: https://patentgrep.com/patent/US12217421B2 ## CPC - G06T 7/0012: {Biomedical image inspection} - G06V 10/82: using neural networks - G06V 40/18: Eye characteristics, e.g. of the iris - G06V 40/193: {Preprocessing; Feature extraction} - G06T 2207/30041: Eye; Retina; Ophthalmic - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Disclosed herein are system, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for tracking a point of gaze of a subject in a manner that incorporates automatic eye tracking calibration. In an embodiment, video is obtained of the subject viewing content presented by a display. The video is analyzed to determine a series of time points at which a gaze of the subject is dwelling on a visual target and, for each time point in the series, a set of eye feature values. A location of a stimulus presented by the display is also obtained for each time point in the series. This information is utilized to determine a mapping that maps a set of eye feature values to a point of gaze. The mapping is used to map a set of eye features obtained by analyzing the video to the point of gaze of the subject. ## Claims ### Claim 1 A computer-implemented method for tracking a point of gaze of a subject, comprising: obtaining, by at least one computer processor, video of at least a portion of a face of the subject while the subject is viewing content of a test sequence presented within a display area of a display, wherein an integrated calibration process is performed during the test sequence, the integrated calibration process comprising: determining, by analyzing the video a series of time points at which a gaze of the subject is dwelling on a visual target and, for at least each time point in the series of time points, a corresponding set of eye features; after determining the series of time points, obtaining, for each of the time points in the series of time points, a location of a visual stimulus presented within the display area of the display; and determining a mapping that maps a set of eye features to a point of gaze, wherein the determining is performed based at least on (i) the set of eye features for each of the time points in the series of time points and (ii) the location of the visual stimulus within the display area of the display for each of the time points in the series of time points; and using the mapping to respectively map one or more sets of eye features to one or more points of gaze of the subject. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein: the one or more sets of eye features obtained by analyzing the video comprises one or more sets of right eye features and the one or more points of gaze of the subject comprises one or more points of gaze of a right eye of the subject; or the one or more sets of eye features obtained by analyzing the video comprises one or more sets of left eye features and the one or more points of gaze of the subject comprises one or more points of gaze of a left eye of the subject. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising: detecting a neurological or mental health condition of the subject based at least on the one or more points of gaze of the subject. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein obtaining the video of the at least a portion of the face of the subject while the subject is viewing the content of the test sequence presented within the display area of the display comprises: obtaining the video of the at least a portion of the face of the subject while the subject is performing a directed oculometric task in response to the visual stimulus presented within the display area of the display. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein the directed oculometric task comprises one of: a saccade test; a smooth pursuit test; or a long fixation test. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein each set of eye features comprises one or more of: a pupil location relative to an eye opening; eye corner locations of an eye; or a length and an orientation of minor and major elliptical axes of an iris of an eye. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein determining the mapping comprises: performing a regression analysis to determine the mapping. ### Claim 8 (depends on 7) The computer-implemented method of claim 7, wherein performing the regression analysis to determine the mapping comprises one of: performing a linear regression analysis; performing a polynomial regression analysis; or performing a decision tree regression analysis. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein analyzing the video to determine, for each of the time points in the series of time points, the set of eye features comprises: extracting an eye image from a frame of the video corresponding to the time point in the series of time points; and providing the eye image to a neural network that, based at least on the eye image, outputs an estimated point of gaze for the time point in the series of time points. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the integrated calibration process further comprising: analyzing the video to determine one or more additional time points at which the gaze of the subject is dwelling on a visual target and, for each of the one or more additional time points, a set of corresponding eye features; obtaining, for each of the one or more additional time points, a location of the visual stimulus within the display area of the display; and recalculating the mapping based at least on (i) the set of eye features for each of the one or more additional time points and (ii) the location of the visual stimulus within the display area of the display for each of the one or more additional time points. ### Claim 11 A system for measuring tracking a point of gaze of a subject, comprising: one or more memories; and at least one processor each coupled to at least one of the memories and configured to perform operations comprising: obtaining video of at least a portion of a face of the subject while the subject is viewing content of a test sequence presented within a display area of a display, wherein an integrated calibration process is performed during the test sequence, the integrated calibration process comprising: determining, by analyzing the video a series of time points at which a gaze of the subject is dwelling on a visual target and, for at least each time point in the series of time points, a corresponding set of eye features; after determining the series of time points, obtaining, for each of the time points in the series of time points, a location of a visual stimulus presented within the display area of the display; and determining a mapping that maps a set of eye features to a point of gaze, wherein the determining is performed based at least on (i) the set of eye features for each of the time points in the series of time points and (ii) the location of the visual stimulus within the display area of the display for each of the time points in the series of time points; and using the mapping to respectively map one or more sets of eye features to one or more points of gaze of the subject. ### Claim 12 (depends on 11) The system of claim 11, wherein: the one or more sets of eye features obtained by analyzing the video comprises one or more sets of right eye features and the one or more points of gaze of the subject comprises one or more points of gaze of a right eye of the subject; or the one or more sets of eye features obtained by analyzing the video comprises one or more sets of left eye features and the one or more points of gaze of the subject comprises one or more points of gaze of a left eye of the subject. ### Claim 13 (depends on 11) The system of claim 11, wherein the operations further comprise: detecting a neurological or mental health condition of the subject based at least on the one or more points of gaze of the subject. ### Claim 14 (depends on 11) The system of claim 11, wherein obtaining the video of the at least a portion of the face of the subject while the subject is viewing the content of the test sequence presented within the display area of the display comprises: obtaining the video of the at least a portion of the face of the subject while the subject is performing a directed oculometric task in response to the visual stimulus presented within the display area of the display. ### Claim 15 (depends on 14) The system of claim 14, wherein the directed oculometric task comprises one of: a saccade test; a smooth pursuit test; or a long fixation test. ### Claim 16 (depends on 11) The system of claim 11, wherein each set of eye features comprises one or more of: a pupil location relative to an eye opening; eye corner locations of an eye; or a length and an orientation of minor and major elliptical axes of an iris of an eye. ### Claim 17 (depends on 11) The system of claim 11, wherein determining the mapping comprises: performing a regression analysis to determine the mapping. ### Claim 18 (depends on 17) The system of claim 17, wherein performing the regression analysis to determine the mapping comprises one of: performing a linear regression analysis; performing a polynomial regression analysis; or performing a decision tree regression analysis. ### Claim 19 (depends on 11) The system of claim 11, wherein analyzing the video to determine, for each of the time points in the series of time points, the set of eye features comprises: extracting an eye image from a frame of the video corresponding to the time point in the series of time points; and providing the eye image to a neural network that, based at least on the eye image, outputs an estimated point of gaze for the time point in the series of time points. ### Claim 20 (depends on 11) The system of claim 11, wherein the integrated calibration process further comprises: analyzing the video to determine one or more additional time points at which the gaze of the subject is dwelling on a visual target and, for each of the one or more additional time points, a set of corresponding eye features; obtaining, for each of the one or more additional time points, a location of the visual stimulus within the display area of the display; and recalculating the mapping based at least on (i) the set of eye features for each of the one or more additional time points and (ii) the location of the visual stimulus within the display area of the display for each of the one or more additional time points. ### Claim 21 A non-transitory computer-readable medium having instructions stored thereon that, when executed by at least one computer processor, cause the at least one computer processor to perform operations for tracking a point of gaze of a subject, the operations comprising: obtaining video of at least a portion of a face of the subject while the subject is viewing content of a test sequence presented within a display area of a display, wherein an integrated calibration process is performed during the test sequence, the integrated calibration process comprising: determining, by analyzing the video a series of time points at which a gaze of the subject is dwelling on a visual target and, for at least each time point in the series of time points, a corresponding set of eye features; after determining the series of time points, obtaining, for each of the time points in the series of time points, a location of a visual stimulus presented within the display area of the display; and determining a mapping that maps a set of eye features to a point of gaze, wherein the determining is performed based at least on (i) the set of eye features for each of the time points in the series of time points and (ii) the location of the visual stimulus within the display area of the display for each of the time points in the series of time points; and using the mapping to respectively map one or more sets of eye features to one or more points of gaze of the subject. ### Claim 22 (depends on 21) The non-transitory computer-readable medium of claim 21, wherein: the one or more sets of eye features obtained by analyzing the video comprises one or more sets of right eye features and the one or more points of gaze of the subject comprises one or more points of gaze of a right eye of the subject; or the one or more sets of eye features obtained by analyzing the video comprises one or more sets of left eye features and the one or more points of gaze of the subject comprises one or more points of gaze of a left eye of the subject. ### Claim 23 (depends on 21) The non-transitory computer-readable medium of claim 21, wherein the operations further comprise: detecting a neurological or mental health condition of the subject based at least on the one or more points of gaze of the subject. ### Claim 24 (depends on 21) The non-transitory computer-readable medium of claim 21, wherein obtaining the video of the at least a portion of the face of the subject while the subject is viewing the content of the test sequence presented within the display area of the display comprises: obtaining the video of the at least a portion of the face of the subject while the subject is performing a directed oculometric task in response to the visual stimulus presented within the display area of the display. ### Claim 25 (depends on 24) The non-transitory computer-readable medium of claim 24, wherein the directed oculometric task comprises one of: a saccade test; a smooth pursuit test; or a long fixation test. ### Claim 26 (depends on 21) The non-transitory computer-readable medium of claim 21, wherein each set of eye features comprises one or more of: a pupil location relative to an eye opening; eye corner locations of an eye; or a length and an orientation of minor and major elliptical axes of an iris of an eye. ### Claim 27 (depends on 21) The non-transitory computer-readable medium of claim 21, wherein determining the mapping comprises: performing a regression analysis to determine the mapping. ### Claim 28 (depends on 27) The non-transitory computer-readable medium of claim 27, wherein performing the regression analysis to determine the mapping comprises one of: performing a linear regression analysis; performing a polynomial regression analysis; or performing a decision tree regression analysis. ### Claim 29 (depends on 21) The non-transitory computer-readable medium of claim 21, wherein analyzing the video to determine, for each of the time points in the series of time points, the set of eye features comprises: extracting an eye image from a frame of the video corresponding to the time point in the series of time points; and providing the eye image to a neural network that, based at least on the eye image, outputs an estimated point of gaze for the time point in the series of time points. ### Claim 30 (depends on 21) The non-transitory computer-readable medium of claim 21, wherein the integrated calibration process comprises: analyzing the video to determine one or more additional time points at which the gaze of the subject is dwelling on a visual target and, for each of the one or more additional time points, a set of corresponding eye features; obtaining, for each of the one or more additional time points, a location of the visual stimulus within the display area of the display; and recalculating the mapping based at least on (i) the set of eye features for each of the one or more additional time points and (ii) the location of the visual stimulus within the display area of the display for each of the one or more additional time points.