# US 12,118,825 B2 · Obtaining high-resolution oculometric parameters > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,825 B2 - Document ID: US12118825B2 - Status: Granted - Kind: B2 - Application number: 17/722,095 - Filed: 2022-04-15 - Published: 2022-11-03 - Granted: 2024-10-15 - Earliest claimed: 2021-05-03 - Assignees: NeuraLight Ltd. - Applicants: NeuraLight Ltd. - Inventors: Edmund Ben-Ami; Micha Yochanan Breakstone; Rotem Zvi Bar-Or; Vladimir Anisimov - Web page: https://patentgrep.com/patent/US12118825B2 ## CPC - G06V 40/18: Eye characteristics, e.g. of the iris - A61B 3/0025: {characterised by electronic signal processing, e.g. eye models} - A61B 3/113: for determining or recording eye movement - A61B 5/163: {by tracking eye movement, gaze, or pupil change} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/09: Supervised learning - G06T 7/0012: {Biomedical image inspection} - G06T 7/0016: {involving temporal comparison} - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06V 10/82: using neural networks - G06N 3/084: Backpropagation, e.g. using gradient descent - G06T 2207/10016: Video; Image sequence - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30041: Eye; Retina; Ophthalmic - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Disclosed are systems and methods for extracting high resolution oculometric parameters. A video stream having a video of a face of a user is processed to obtain a set of oculometric parameters, such as eyelid data, iris data (e.g., iris translation, iris radius and iris rotation), and pupil data (e.g., pupil center and pupil radius) at a first resolution. A deconvolution process is performed on the video stream to improve accuracy or resolution of the oculometric parameters, based on stimulus information of a video stimulus displayed on a client device associated with the user, environment data of an environment in which the user is located, device data of the client device, etc. The oculometric parameters are then processed using a prediction model that is trained based on high resolution oculometric parameters obtained using eye tracking devices to predict oculometric parameters at a resolution greater than the first resolution. ## Claims ### Claim 1 A method comprising: obtaining a video stream from a client device associated with a user, the video stream including a video of a face of the user; providing as input the video stream to a first prediction model to obtain a first set of oculometric parameters of an eye of the user, the first set of oculometric parameters including a first pupil center, first iris data, and first eyelid data, wherein the first set of oculometric parameters are obtained at a first resolution; deconvolving the video stream based on the first eyelid data, the first pupil center, stimulus data and environment data to obtain adjusted eyelid data, an adjusted pupil center and an adjusted pupil radius as part of the first set of oculometric parameters, the stimulus data including spatio-temporal information and optical information of a video stimulus presented on a display associated with the client device, the environment data including light conditions in an environment the client device is in; and providing as input the first set of oculometric parameters to a second prediction model to obtain a second set of oculometric parameters at a second resolution, wherein the second set of oculometric parameters includes (a) a second pupil center, (b) a second pupil radius, and (c) second iris data including an iris radius, an iris translation, and an iris rotation, and wherein the second resolution is greater than the first resolution. ### Claim 2 (depends on 1) The method of claim 1, wherein the first resolution is measured in terms of a pixel or other lower resolution value and the second resolution is measured in terms of a sub-pixel value. ### Claim 3 (depends on 1) The method of claim 1, wherein providing the input to the second prediction model includes: training the second prediction model using a plurality of datasets to predict the second set of oculometric parameters at the second resolution, wherein each dataset includes a video of a face of a specified user, a first set of values corresponding to the first set of oculometric parameters obtained at the first resolution, and a set of labels having a second set of values corresponding to the second set of oculometric parameters obtained at the second resolution. ### Claim 4 (depends on 3) The method of claim 3, wherein the second set of values corresponding to the second set of oculometric parameters is obtained using an eye-tracking device that is configured to obtain the second set of oculometric parameters at the second resolution for a video of the face of the specified user. ### Claim 5 (depends on 1) The method of claim 1, wherein deconvolving the video stream includes: determining a point spread function of a deconvolution process based on the first eyelid data, the first pupil center, the stimulus data and the environment data; and deconvolving the video stream using the point spread function to obtain the adjusted eyelid data, the adjusted pupil center and the adjusted pupil radius. ### Claim 6 (depends on 5) The method of claim 5, wherein deconvolving the video stream includes determining the adjusted eyelid data, the adjusted pupil center and the adjusted pupil radius based on a maximal likelihood of a prediction of a location of the first pupil center and an eyelid boundary. ### Claim 7 (depends on 5) The method of claim 5, wherein determining the adjusted eyelid data includes: processing the first eyelid data using a maximal likelihood estimation curve fitting operation to obtain a parabolic fit of an eyelid of the eye, and obtaining a plurality of coordinates along the parabolic fit as the adjusted eyelid data. ### Claim 8 (depends on 6) The method of claim 6, wherein determining the adjusted pupil center includes: determining a plurality of locations for the first pupil center based on the maximal likelihood, generating a shape of a pupil for each of the plurality of locations, based on a set of constraints and a set of assumptions, assigning a score for each of the plurality of shapes of the pupil, selecting a specified shape of the plurality of shapes based on a specified score of the specified shape satisfying a criterion, and determining a center and radius of the specified shape as the adjusted pupil center and the adjusted pupil radius, respectively. ### Claim 9 (depends on 1) The method of claim 1, wherein providing the video stream to the first prediction model to obtain the first set of oculometric parameters includes: prior to inputting the video stream to the first prediction model, processing the video stream to a format suitable for the first prediction model to extract the first set of oculometric parameters. ### Claim 10 (depends on 9) The method of claim 9, wherein processing the video stream includes: reducing noise, adjusting color and brightness of the video, and improving a resolution of the video. ### Claim 11 (depends on 1) The method of claim 1, wherein providing the video stream to the first prediction model to obtain the first set of oculometric parameters further includes: obtaining user data associated with the user, the user data including optometric data of the eye of the user, and applying corrections to the first set of oculometric parameters based on the user data. ### Claim 12 (depends on 1) The method of claim 1, wherein the first iris data includes (a) a plurality of coordinates representative of a shape of an iris, and (b) coordinates of a first iris translation. ### Claim 13 (depends on 12) The method of claim 12, wherein providing the first set of oculometric parameters to the second prediction model includes: processing the first iris data using a maximal likelihood estimation-based curve fitting operation to obtain an oval fit of the shape of the iris; and determining a first iris rotation and a first iris radius based on the oval fit. ### Claim 14 (depends on 1) The method of claim 1, wherein the first eyelid data includes a plurality of coordinates of a shape representative of eyelids of the eye. ### Claim 15 (depends on 1) The method of claim 1, wherein the first set of oculometric parameters are obtained as time series data, wherein a first set of values of the first set of oculometric parameters are obtained for a first moment in the video and a second set of values of the first set of oculometric parameters are obtained for a second moment in the video. ### Claim 16 (depends on 1) The method of claim 1, wherein the first set of oculometric parameters are obtained in a plurality of coordinate systems. ### Claim 17 (depends on 16) The method of claim 16, wherein the plurality of coordinate systems includes a device coordinate system in which an origin is located at a center of a display associated with the client device. ### Claim 18 (depends on 16) The method of claim 16, wherein the plurality of coordinate systems includes a head coordinate system in which an origin is located at a center between eyes as projected on a plane defining the face of the user by displacement in a z-direction. ### Claim 19 (depends on 1) The method of claim 1 further comprising: determining an additional set of oculometric parameters based on the second set of oculometric parameters, the additional set of oculometric parameters including at least one of a pupil visible fraction, an iris visible fraction, a pupil coverage asymmetry, or an iris coverage asymmetry. ### Claim 20 A non-transitory computer-readable medium having instructions stored thereon that, when executed by a computer, cause the computer to execute a method, the method comprising: obtaining a video stream from a client device associated with a user, the video stream including a video of a face of the user; providing as input the video stream to a first prediction model to obtain a first set of oculometric parameters of an eye of the user, the first set of oculometric parameters including a first pupil center, first iris data, and first eyelid data; deconvolving the video stream based on the first eyelid data, the first pupil center, stimulus data and environment data to obtain adjusted eyelid data, an adjusted pupil center and an adjusted pupil radius as part of the first set of oculometric parameters, the stimulus data including spatio-temporal information and optical information of a video stimulus presented on a display associated with the client device, the environment data including light conditions in an environment the client device is in; and providing as input the first set of oculometric parameters to a second prediction model to obtain a second set of oculometric parameters at a greater resolution than the first set of oculometric parameters, wherein the second set of oculometric parameters includes a second pupil center, a second pupil radius, an iris radius, an iris translation, and an iris rotation. ### Claim 21 (depends on 20) The computer-readable medium of claim 20, wherein providing the input to the second prediction model includes: training the second prediction model using a plurality of datasets to predict the second set of oculometric parameters, wherein each dataset includes a video of a face of a specified user, a first set of values corresponding to the first set of oculometric parameters obtained at a specified resolution, and a set of labels having a second set of values corresponding to the second set of oculometric parameters obtained at a resolution greater than the specified resolution. ### Claim 22 (depends on 21) The computer-readable medium of claim 21, wherein the second set of values corresponding to the second set of oculometric parameters is obtained using an eye-tracking device that is configured to obtain the second set of oculometric parameters at a greater resolution than the specified resolution. ### Claim 23 (depends on 20) The computer-readable medium of claim 20, wherein deconvolving the video stream includes: determining a point spread function of a deconvolution process based on the first eyelid data, the first pupil center, the stimulus data and the environment data; and deconvolving the video stream using the point spread function to obtain the adjusted eyelid data, the adjusted pupil center and the adjusted pupil radius. ### Claim 24 (depends on 23) The computer-readable medium of claim 23, wherein deconvolving the video stream includes determining the adjusted eyelid data, the adjusted pupil center and the adjusted pupil radius based on a maximal likelihood of a prediction of a location of the first pupil center and an eyelid boundary. ### Claim 25 (depends on 24) The computer-readable medium of claim 24, wherein determining the adjusted eyelid data includes: processing the first eyelid data using a maximal likelihood estimation curve fitting operation to obtain a parabolic fit of an eyelid of the eye, and obtaining a plurality of coordinates along the parabolic fit as the adjusted eyelid data. ### Claim 26 (depends on 24) The computer-readable medium of claim 24, wherein determining the adjusted pupil center includes: determining a plurality of locations for the first pupil center based on the maximal likelihood, generating a shape of a pupil for each of the plurality of locations, based on a set of constraints and a set of assumptions, assigning a score for each of the plurality of shapes of the pupil, selecting a specified shape of the plurality of shapes based on a specified score of the specified shape satisfying a criterion, and determining a center and radius of the specified shape as the adjusted pupil center and the pupil radius, respectively. ### Claim 27 (depends on 20) The computer-readable medium of claim 20, wherein the first set of oculometric parameters are obtained as time series data, wherein a first set of values of the first set of oculometric parameters are obtained for a first moment in the video and a second set of values of the first set of oculometric parameters are obtained for a second moment in the video. ### Claim 28 (depends on 20) The computer-readable medium of claim 20 further comprising: determining an additional set of oculometric parameters based on the second set of oculometric parameters, the additional set of oculometric parameters including at least one of a pupil visible fraction, an iris visible fraction, a pupil coverage asymmetry, or an iris coverage asymmetry. ### Claim 29 A system, comprising: a memory storing a set of instructions; and a processor configured to execute the set of instructions to cause the system to perform a method of: obtaining a video stream from a client device associated with a user, the video stream including a video of a face of the user; training a first prediction model using a first plurality of datasets to output a first set of oculometric parameters of an eye of the user at a first resolution, the first set of oculometric parameters including a first pupil center, first iris data, and first eyelid data, wherein each dataset of the first plurality of datasets includes a video of a face of a specified user and a set of oculometric parameters obtained at the first resolution; deconvolving the video stream based on stimulus data and environment data to obtain adjusted eyelid data, an adjusted pupil center and an adjusted pupil radius as part of the first set of oculometric parameters; and training a second prediction model using a second plurality of datasets to output a second set of oculometric parameters at a second resolution greater than the first resolution, wherein the second set of oculometric parameters includes a second pupil center, a second pupil radius, an iris radius, an iris translation, and an iris rotation, wherein each dataset includes a video of a face of a specified user, a first set of values corresponding to the first set of oculometric parameters obtained at the first resolution, and a set of labels having a second set of values corresponding to the second set of oculometric parameters obtained at the second resolution.