# US 2025/0148827 A1 · Correcting pupil center shift to compute gaze > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2025/0148827 A1 - Document ID: US20250148827A1 - Status: Published application - Kind: A1 - Application number: 18/920,368 - Filed: 2024-10-18 - Published: 2025-05-08 - Earliest claimed: 2023-11-02 - Assignees: Snap Inc. - Applicants: Snap Inc. - Inventors: Candace Elise Peacock; Ramzi Zahreddine - Web page: https://patentgrep.com/patent/US20250148827A1 ## CPC - G06V 40/18: Eye characteristics, e.g. of the iris - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Systems and methods are provided for computing gaze. The systems and methods access an uncorrected gaze vector computed based on a center point of a pupil of a user. The systems and methods process an image of the pupil by a machine learning model to predict an estimated error in a gaze vector, the machine learning model trained to establish a relationship between a plurality of ground truth gaze vectors and uncorrected gaze vectors for a plurality of pupil parameters, the pupil parameters including diameters, gaze angles, or gaze eccentricities. The systems and methods generate a corrected gaze vector by applying the estimated error in the gaze vector predicted by the machine learning model to the uncorrected gaze vector that has been computed based on the center point of the pupil of the user. ## Claims ### Claim 1 A method comprising: accessing an uncorrected gaze vector computed based on a center point of a pupil of a user; processing an image of the pupil by a machine learning model to predict an estimated error in a gaze vector, the machine learning model trained to establish a relationship between a plurality of ground truth gaze vectors and uncorrected gaze vectors for a plurality of pupil parameters, the pupil parameters including diameters, gaze angles, or gaze eccentricities; and generating a corrected gaze vector by applying the estimated error in the gaze vector predicted by the machine learning model to the uncorrected gaze vector that has been computed based on the center point of the pupil of the user. ### Claim 2 (depends on 1) The method of claim 1, further comprising: performing one or more augmented reality operations based on the corrected gaze vector. ### Claim 3 (depends on 2) The method of claim 2, further comprising: identifying an object depicted in a display of an augmented reality device that corresponds to the corrected gaze vector; and performing the one or more augmented reality operations in relation to the identified object. ### Claim 4 (depends on 1) The method of claim 1, wherein the machine learning model is further trained to establish the relationship between the plurality of ground truth gaze vectors and the uncorrected gaze vectors based on gaze eccentricity. ### Claim 5 (depends on 1) The method of claim 1, further comprising: training a first component of the machine learning model to predict a first estimated error in a first angular component; training a second component of the machine learning model to predict a second estimated error in a second angular component; and training a third component of the machine learning model to predict a third estimated error in a third angular component. ### Claim 6 (depends on 5) The method of claim 5, wherein the first, second, and third components are trained separately and independently of each other. ### Claim 7 (depends on 5) The method of claim 5, wherein the first, second, and third components are trained for each eye of a plurality of eyes of the user to enable corrections to be applied to a combined uncorrected gaze vector associated with left and right eyes of a user or to be applied separately to a first uncorrected gaze vector associated with the left eye and a second uncorrected gaze vector associated with the right eye. ### Claim 8 (depends on 5) The method of claim 5, further comprising: accessing a plurality of training data comprising the uncorrected gaze vectors for the plurality of pupil diameters and the plurality of ground truth gaze vectors associated with the uncorrected gaze vectors for the plurality of pupil diameters; obtaining a first batch of the training data comprising a first uncorrected gaze vector for a first pupil diameter; processing the first pupil diameter by the machine learning model to predict a first estimated error in the first uncorrected gaze vector; computing a ground truth error between the first uncorrected gaze vector and a first ground truth gaze vector associated with the first uncorrected gaze vector; computing a deviation between the ground truth error and the first estimated error; and updating one or more parameters of the machine learning model based on the computed deviation. ### Claim 9 (depends on 8) The method of claim 8, wherein the first estimated error comprises a plurality of estimated errors for different angular components, further comprising: computing a plurality of ground truth errors each associated with a different angular component of the first uncorrected gaze vector and the first ground truth gaze vector; and computing a plurality of deviations between the plurality of ground truth errors and the plurality of estimated errors for different angular components. ### Claim 10 (depends on 9) The method of claim 9, wherein the one or more parameters of the machine learning model are updated based on the plurality of deviations. ### Claim 11 (depends on 8) The method of claim 8, wherein the ground truth error is computed as a function of the first pupil diameter, a square of the first pupil diameter, and eccentricity of reported gaze associated with the first pupil diameter. ### Claim 12 (depends on 8) The method of claim 8, further comprising generating at least a portion of the plurality of training data by performing calibration operations comprising: presenting a stimulus on a display, the stimulus having a known location; determining a training pupil diameter; computing a training uncorrected gaze vector based on the detected pupil diameter; computing a known gaze vector based on a correlation between the detected pupil diameter and the known location of the stimulus; and storing the training uncorrected gaze vector and the detected pupil diameter in association with the known gaze vector as the portion of the plurality of training data. ### Claim 13 (depends on 12) The method of claim 12, further comprising: repeating the calibration operations for a plurality of background light intensities. ### Claim 14 (depends on 13) The method of claim 13, wherein the plurality of background light intensities ranges from 0.5 nits to 2000 nits to capture a full range of pupil diameters expected in unconstrained settings. ### Claim 15 (depends on 1) The method of claim 1, wherein the uncorrected gaze vector is computed as a function of corneal reflection and a center position of the pupil. ### Claim 16 (depends on 1) The method of claim 1, further comprising detecting changes to the detected point of the pupil based on different environmental conditions, the change in the detected point causing errors in the computed uncorrected gaze vector, the errors comprising at least one of error in gaze position, direction, or vergence depth estimation. ### Claim 17 (depends on 1) The method of claim 1, further comprising adding the estimated error in the gaze vector to the uncorrected gaze vector to generate the corrected gaze vector. ### Claim 18 A system comprising: at least one storage device; and at least one processor coupled to the at least one storage device and configured to perform operations comprising: accessing an uncorrected gaze vector computed based on a center point of a pupil of a user; processing an image of the pupil by a machine learning model to predict an estimated error in a gaze vector, the machine learning model trained to establish a relationship between a plurality of ground truth gaze vectors and uncorrected gaze vectors for a plurality of pupil parameters, the pupil parameters including diameters, gaze angles, or gaze eccentricities; and generating a corrected gaze vector by applying the estimated error in the gaze vector predicted by the machine learning model to the uncorrected gaze vector that has been computed based on the center point of the pupil of the user. ### Claim 19 (depends on 18) The system of claim 18, wherein the operations comprise: performing one or more augmented reality operations based on the corrected gaze vector. ### Claim 20 A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising: accessing an uncorrected gaze vector computed based on a center point of a pupil of a user; processing an image of the pupil by a machine learning model to predict an estimated error in a gaze vector, the machine learning model trained to establish a relationship between a plurality of ground truth gaze vectors and uncorrected gaze vectors for a plurality of pupil parameters, the pupil parameters including diameters, gaze angles, or gaze eccentricities; and generating a corrected gaze vector by applying the estimated error in the gaze vector predicted by the machine learning model to the uncorrected gaze vector that has been computed based on the center point of the pupil of the user.