# US 11,308,321 B2 · Method and system for 3D cornea position estimation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,321 B2 - Document ID: US11308321B2 - Status: Granted - Kind: B2 - Application number: 16/915,233 - Filed: 2020-06-29 - Published: 2021-01-14 - Granted: 2022-04-19 - Earliest claimed: 2019-06-28 - Assignees: Tobii AB - Applicants: Tobii AB - Inventors: David Masko; Magnus Ivarsson; Niklas Ollesson; Anna Redz - Web page: https://patentgrep.com/patent/US11308321B2 ## CPC - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G06T 7/277: involving stochastic approaches, e.g. using Kalman filters - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 7/73: using feature-based methods - G06V 20/64: Three-dimensional [3D] objects - G06V 40/19: Sensors therefor - G06V 40/193: {Preprocessing; Feature extraction} - G02B 27/0172: {characterised by optical features} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - G06T 2200/04: involving 3D image data - G06T 2207/30041: Eye; Retina; Ophthalmic - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract There is provided a method, system, and non-transitory computer-readable storage medium for performing three-dimensional, 3D, position estimation for the cornea center of an eye of a user, using a remote eye tracking system, wherein the position estimation is reliable and robust also when the cornea center moves over time in relation to an imaging device associated with the eye tracking system. This is accomplished by generating, using, and optionally also updating, a cornea movement filter, CMF, in the cornea center position estimation. ## Claims ### Claim 1 A method for performing three-dimensional, 3D, position estimation for a cornea center of an eye of a user, using a remote eye tracking system, when the cornea center moves over time in relation to an imaging device associated with the eye tracking system, the method comprising: generating, using processing circuitry associated with the eye tracking system, a cornea movement filter, CMF, comprising an estimated initial 3D position and an estimated initial 3D velocity of the cornea center of the eye at a first time instance, wherein the cornea movement filter, CMF, is a Kalman filter; predicting, using the processing circuitry: a first two-dimensional, 2D, glint position in an image captured at a second time instance by applying the cornea movement filter, CMF, wherein the predicted first glint position represents a position where a first glint is predicted to be generated by a first illuminator associated with the eye tracking system; and a second 2D glint position in the image captured at the second time instance by applying the cornea movement filter, CMF, wherein the predicted second glint position represents a position where a glint is predicted to be generated by a second illuminator associated with the eye tracking system; performing image processing, using the processing circuitry, including the steps of: identifying at least one first candidate glint in a first image captured by the imaging device at the second time instance wherein the first image comprises at least part of the cornea of the eye and at least one glint generated by the first illuminator; and identifying at least one second candidate glint in the first image or in a second image captured by the imaging device at a third time instance and comprising at least part of the cornea of the eye and at least one glint generated by the second illuminator; selecting, using the processing circuitry, a pair of a first and second candidate glint, out of all possible pairs of a first candidate glint and a second candidate glint, that has a highest probability of corresponding to the predicted first and second glint positions, based on a probability estimation function; estimating a current 3D position of the cornea, using the processing circuitry, based on positions of the selected pair of first and second candidate glints; and updating the cornea movement filter, CMF, using the processing circuitry, by setting the estimated initial 3D position, of the cornea center to the current 3D position of the cornea center; and determining a current estimated 3D velocity of the cornea center based on the estimated initial 3D position of the cornea center and the current 3D position of the cornea center; and setting the estimated initial 3D velocity of the cornea center to the current estimated 3D velocity of the cornea center. ### Claim 2 (depends on 1) The method of claim 1, comprising performing the method steps of claim 1 repeatedly. ### Claim 3 (depends on 1) The method of claim 1, wherein updating the cornea movement filter, CMF, is done using an Extended Kalman Filter equation and a linearized version of cornea-to-glint-pair equations. ### Claim 4 (depends on 1) The method of claim 1, wherein selecting, using the processing circuitry, the pair of a first and second candidate glint, out of all possible pairs of a first candidate glint and a second candidate glint, that has the highest probability of corresponding to the predicted first and second glint positions comprises: determining a probability that the positions of the first and second candidate glint in the glint pair correspond to the predicted first and second glint positions P1, P2; determining a closeness of the glint pair to a selected part of the eye; and selecting the glint pair from the possible glint pairs based on a result from the probability determination and the closeness determination. ### Claim 5 (depends on 1) The method of claim 1, the method further comprising, if the at least one second candidate glint is identified in the second image captured by the imaging device at a third time instance compensating for a movement of the at least one second candidate glint from the second time instance to the third time instance by applying the cornea movement filter, CMF, before updating the cornea movement filter, CMF. ### Claim 6 (depends on 1) The method of claim 1, further comprising, before updating the cornea movement filter, CMF: comparing the probability of the selected pair of candidate glints to a preset threshold value; and if the probability of the selected pair of candidate glints exceeds the preset threshold value, setting the predicted first and second glint positions to the positions of the selected pair of candidate glints. ### Claim 7 (depends on 1) The method of claim 1, further comprising, the first time the method is performed, setting a respective start value for the estimated initial 3D position and the estimated initial 3D velocity of the cornea center of the eye at the first time instance based on a respective estimated 3D position and velocity of the eye or head of the user. ### Claim 8 (depends on 1) The method of claim 1, further comprising predicting, by the eye tracking system, a gaze of the user based on the estimated current 3D position of the cornea. ### Claim 9 An eye tracking system for performing three-dimensional, 3D, position estimation for a cornea center of an eye of a user when the cornea center moves over time in relation to an imaging device associated with the eye tracking system, the eye tracking system comprising processing circuitry configured to: generate a cornea movement filter, CMF, comprising an estimated initial 3D position and an estimated initial 3D velocity of the cornea center of the eye at a first time instance, wherein the cornea movement filter, CMF, is a Kalman filter; predict a first two-dimensional, 2D, glint position in an image captured at a second time instance by applying the cornea movement filter, CMF, wherein the predicted first glint position represents a position where a first glint is predicted to be generated by a first illuminator associated with the eye tracking system; and predict a second 2D glint position in the image captured at the second time instance by applying the cornea movement filter, CMF, wherein the predicted second glint position represents a position where a glint is predicted to be generated by a second illuminator associated with the eye tracking system; perform image processing to: identify at least one first candidate glint in a first image captured by the imaging device at the second time instance, wherein the first image comprises at least part of the cornea of the eye and at least one glint generated by the first illuminator; and identify at least one second candidate glint in the first image or in a second image captured by the imaging device at a third time instance and comprising at least part of the cornea of the eye and at least one glint generated by the second illuminator; select a pair of a first and second candidate glint, out of all possible pairs of a first candidate glint and a second candidate glint, that has a highest probability of corresponding to the predicted first and second glint positions, based on a probability estimation function; estimate a current 3D position of the cornea based on positions of the selected pair of first and second candidate glints; and update the cornea movement filter, CMF, by setting the estimated initial 3D position of the cornea center to the current 3D position of the cornea center; and determine a current estimated 3D velocity of the cornea center based on the estimated initial 3D position of the cornea center and the current 3D position of the cornea center; and set the estimated initial 3D velocity of the cornea center to the current estimated 3D velocity of the cornea center. ### Claim 10 (depends on 9) The eye tracking system of claim 9, wherein the processing circuitry is configured to perform the steps repeatedly. ### Claim 11 (depends on 9) The eye tracking system of claim 9, wherein the processing circuitry is configured to select the pair of a first and second candidate glint, out of all possible pairs of a first candidate glint and a second candidate glint, that has the highest probability of corresponding to the predicted first and second glint positions, by: determining a probability that the positions of the first and second candidate glint in the glint pair correspond to the predicted first and second glint positions; determine a closeness of the glint pair to a selected part of the eye; and selecting the glint pair from the possible glint pairs based on a result from the probability determination and the closeness determination. ### Claim 12 (depends on 9) The eye tracking system of claim 9, wherein the processing circuitry is further configured to: if the at least one second candidate glint is identified in the second image captured by the imaging device at a third time instance, compensating for a movement of the at least one second candidate glint from the second time instance to the third time instance by applying the cornea movement filter, CMF, before updating the cornea movement filter, CMF. ### Claim 13 (depends on 9) The eye tracking system of claim 9, wherein the processing circuitry is further configured to, before updating the cornea movement filter, CMF: compare the probability of the selected pair of candidate glints to a preset threshold value; and if the probability of the selected pair of candidate glints exceeds the preset threshold value, setting the predicted first and second glint positions to the positions of the selected pair of candidate glints. ### Claim 14 (depends on 9) The eye tracking system of claim 9, wherein the processing circuitry is further configured to, the first time the method is performed, set a respective start value for the estimated initial 3D position and the estimated initial 3D velocity of the cornea center of the eye at the first time instance based on a respective estimated 3D position and velocity of the eye or head of the user. ### Claim 15 (depends on 9) The eye tracking system of claim 9, wherein the eye tracking system is further configured to predict a gaze of the user based on the estimated current 3D position of the cornea. ### Claim 16 A non-transitory computer-readable storage medium storing instructions which, when executed by processing circuitry of an eye tracking system, cause the eye tracking system to: generate a cornea movement filter, CMF, comprising an estimated initial 3D position and an estimated initial 3D velocity of a cornea center of an eye at a first time instance, wherein the cornea movement filter, CMF, is a Kalman filter; and predict a first two-dimensional, 2D, glint position in an image captured at a second time instance by applying the cornea movement filter, CMF, wherein the predicted first glint position represents a position where a first glint is predicted to be generated by a first illuminator associated with the eye tracking system; predict a second 2D glint position in the image captured at the second time instance by applying the cornea movement filter, CMF, wherein the predicted second glint position represents a position where a glint is predicted to be generated by a second illuminator associated with the eye tracking system; perform image processing to: identify at least one first candidate glint in a first image captured by an imaging device at the second time instance wherein the first image comprises at least part of the cornea of the eye and at least one glint generated by the first illuminator; and identify at least one second candidate glint in the first image or in a second image captured by the imaging device at a third time instance and comprising at least part of the cornea of the eye and at least one glint generated by the second illuminator; select a pair of a first and second candidate glint, out of all possible pairs of a first candidate glint and a second candidate glint that has a highest probability of corresponding to the predicted first and second glint positions, based on a probability estimation function; estimate a current 3D position of the cornea based on positions of the selected pair of first and second candidate glints; and update the cornea movement filter, CMF, by setting the estimated initial 3D position of the cornea center to the current 3D position of the cornea center; determine a current estimated 3D velocity of the cornea center based on the estimated initial 3D position of the cornea center and the current 3D position of the cornea center; and set the estimated initial 3D velocity of the cornea center to the current estimated 3D velocity of the cornea center.