# US 11,443,440 B2 · Method of processing images of a users eye, associated systems and computer programs > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,440 B2 - Document ID: US11443440B2 - Status: Granted - Kind: B2 - Application number: 17/039,869 - Filed: 2020-09-30 - Published: 2021-08-19 - Granted: 2022-09-13 - Earliest claimed: 2019-09-30 - Assignees: Tobii AB - Applicants: Tobii AB - Inventors: Mark Ryan; Oscar Lundqvist; Oscar Nyman - Web page: https://patentgrep.com/patent/US11443440B2 ## CPC - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/012: {Head tracking input arrangements} - G06F 18/2193: {based on specific statistical tests} - G06T 7/254: involving subtraction of images - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 40/18: Eye characteristics, e.g. of the iris - G06V 40/193: {Preprocessing; Feature extraction} - G06T 2207/20224: Image subtraction - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A computer-implemented method of selecting a sequence of images of a user's eye for an eye tracking application wherein each image is captured when a stationary stimulus point is displayed to the user, the method comprising: for a plurality of different pairs of the images: comparing the pair of images with each other to determine an image-score that represents a degree of difference between the compared images; and calculating a sequence-score based on the image-scores for the plurality of pairs of images. ## Claims ### Claim 1 A computer-implemented method of selecting a sequence of images of a user's eye for an eye tracking application wherein each image is captured when a stationary stimulus point is displayed to the user, the method comprising, for a plurality of different pairs of the images: comparing the pair of images with each other to determine an image-score that represents a degree of difference between the compared images; and calculating a sequence-score based on the image-scores for the plurality of pairs of images. ### Claim 2 (depends on 1) The method of claim 1, wherein: each of the images in the sequence comprises a gaze-location-identifier that represents the location of the stimulus on a screen that was presented to the user when the image was captured; and the gaze-location-identifier is the same for each image in the sequence of images. ### Claim 3 (depends on 1) The method of claim 1, further comprising controlling how the sequence of images is passed to a downstream eye gaze tracking operation based on the sequence-score. ### Claim 4 (depends on 1) The method of claim 1, further comprising: setting a discard-flag for at least a portion of the sequence of images if the sequence-score is greater than a discard-threshold. ### Claim 5 (depends on 1) The method of claim 1, further comprising: discarding at least a portion of the sequence of images if the sequence-score is greater than a discard-threshold. ### Claim 6 (depends on 4) The method of claim 4, further comprising: setting an inspection-flag for at least a portion of the sequence of images if the sequence-score is greater than an inspection-threshold, wherein the inspection-threshold is less than the discard-threshold; and if the inspection-flag is set, then providing the at least a portion of the sequence of images to a user for manually setting the discard-flag. ### Claim 7 (depends on 4) The method of claim 4, further comprising: performing a downstream eye gaze tracking operation on sequences of images for which the discard-flag is not set, and not performing the eye gaze tracking operation on sequences of images for which the discard-flag is set. ### Claim 8 (depends on 4) The method of claim 4, further comprising: performing a downstream eye gaze tracking operation on the sequence of images irrespective of whether or not the associated discard-flag is set; and aggregating performance evaluation statistics on sequences of images for which the discard-flag is not set, and not aggregating performance evaluation statistics on sequences of images for which the discard-flag is set. ### Claim 9 (depends on 1) The method of claim 1, further comprising: identifying a plurality of spaced apart sub-sequences of images in the sequence of images, wherein each sub-sequence comprises a plurality of consecutive images in the sequence of images that satisfy a sub-sequence-score-threshold; and concatenating the plurality of sub-sequences of images to provide a concatenated-sequence of images; and providing the concatenated-sequence of images to a downstream eye gaze tracking operation. ### Claim 10 (depends on 7) The method of claim 7, wherein the eye gaze tracking operation comprises: a training algorithm for setting one or more trainable parameters of a machine learning model for performing eye gaze tracking on received images; a calibration algorithm; a verification algorithm; and/or a performance evaluation algorithm. ### Claim 11 (depends on 10) The method of claim 10, wherein: each sequence of images comprises a gaze-location-identifier that represents the location of a stimulus on a screen that was presented to the user when the sequence of images was captured; and the eye gaze tracking operation performs its function based on the images and the gaze-location-identifier. ### Claim 12 (depends on 1) The method of claim 1, wherein calculating the sequence-score based on the image-scores comprises: summing the image-scores to provide an accumulated-sequence-score. ### Claim 13 (depends on 1) The method of claim 1, wherein calculating the sequence-score based on the image-scores comprises: setting a peak-sequence-score as the maximum value of the image-scores. ### Claim 14 (depends on 13) The method of claim 13, further comprising: setting a blink-flag for the sequence of images if the peak-sequence-score is greater than a blink-threshold; and if the blink-flag is set, then passing the sequence of images to a downstream eye gaze tracking operation that requires a presence of a blink in the sequence of images; or if the blink-flag is not set, then passing the sequence of images to a downstream eye gaze tracking operation that requires the absence of a blink in the sequence of images. ### Claim 15 (depends on 13) The method of claim 13, further comprising: setting a movement-flag for the sequence of images if the peak-sequence-score is greater than a movement -threshold; and if the movement-flag is set, then passing the sequence of images to a downstream eye gaze tracking operation that requires a presence of a movement in the sequence of images; or if the movement-flag is not set, then passing the sequence of images to a downstream eye gaze tracking operation that requires the absence of movement in the sequence of images. ### Claim 16 (depends on 1) The method of claim 1, wherein the method further comprises: causing a screen to display a stationary stimulus point to the user in accordance with a gaze-location-identifier; and capturing the sequence of images of the user's eye while the screen is displaying the stationary stimulus point to the user. ### Claim 17 (depends on 1) The method of claim 1, wherein: each of the images comprises a plurality of pixel positions, and each pixel position has a pixel-value that represents a brightness or hue of the pixel; and comparing the pair of images comprises: comparing pixel-values at corresponding pixel-positions in the pair of images to determine pixel-value-differences; and determining the image-score as a function of the pixel-value-differences. ### Claim 18 (depends on 1) The method of claim 1, wherein: each of the images comprises: a plurality of pixel positions that represent a subset of pixels from a full-sensor-image, and a crop-identifier that represents a location of the subset of pixels in the full-sensor-image; and comparing the pair of images comprises determining a crop-offset-value as the difference between the crop-identifiers of the two images; the method further comprises: setting a discard-sequence-threshold based on the crop-offset-values; and setting a discard-flag for the sequence of images if the sequence-score is greater than the discard-sequence-threshold. ### Claim 19 A processor configured to: receive a sequence of images of a user's eye for an eye tracking application; for a plurality of different pairs of the images: compare the pair of images with each other to determine an image-score that represents a degree of difference between the compared images; and calculate a sequence-score based on the image-scores for the plurality of pairs of images. ### Claim 20 A non-transitory computer-readable storage medium storing instructions which, when executed by a cause the computer to carry out the steps of: selecting a sequence of images of a user's eye for an eye tracking application wherein each image is captured when a stationary stimulus point is displayed to the user, comparing a plurality of different pairs of the images with each other to determine an image-score that represents a degree of difference between the compared images; and calculating a sequence-score based on the image-scores for the plurality of pairs of images.