# US 2023/0410367 A1 · Method and System for Determining a Current Gaze Direction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2023/0410367 A1 - Document ID: US20230410367A1 - Status: Published application - Kind: A1 - Application number: 18/212,227 - Filed: 2023-06-21 - Published: 2023-12-21 - Earliest claimed: 2022-06-21 - Assignees: Tobii AB - Applicants: Tobii AB - Inventors: PETER BLIXT; MÅRTEN SELIN - Web page: https://patentgrep.com/patent/US20230410367A1 ## CPC - G06T 7/73: using feature-based methods - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06T 7/13: Edge detection - G06T 7/62: of area, perimeter, diameter or volume - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/82: using neural networks - G06V 40/166: {using acquisition arrangements} - G06V 40/19: Sensors therefor - G06V 40/193: {Preprocessing; Feature extraction} - G06V 40/60: Static or dynamic means for assisting the user to position a body part for biometric acquisition - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30041: Eye; Retina; Ophthalmic - G06T 2207/30201: Face - G06T 2207/30208: Marker matrix - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Method for determining calibration information in relation to a digital camera used to depict a user in a system for determining a user gaze point in relation to an entity, the method comprising: receiving from the camera at least one image of a user; performing digital image processing on the image to determine an in-image feature size dimension of a feature of the entity as reflected in an eye of the user; and to determine an in-image limbus size dimension of a limbus of the eye; determining a current distance between the eye and the entity based on said in-image size dimensions, a curvature of the eye cornea, a physical limbus size and a physical feature size; and determining said calibration information in the form of a focal length and/or a field of view of the camera based on said current distance and said physical limbus size dimension. ## Claims ### Claim 1 Method for determining calibration information in relation to a digital camera, the digital camera being used to depict a second user in a system for determining a gaze point of the second user in relation to an entity, the method comprising: receiving from the digital camera at least one calibration image, being a depiction of at least one eye of a first user, wherein the first user may be the same as or different from the second user; performing digital image processing on the calibration image: to locate and determine an in-image feature size dimension of a feature of the entity as reflected in said eye of the first user; to locate a limbus of said eye of the first user; and to determine an in-image limbus size dimension of said limbus; determining a current distance between the eye of the first user and the entity based on said in-image feature size dimension, said in-image limbus size dimension, a curvature of the cornea of the eye of the first user, a physical limbus size dimension of the eye of the first user and a physical size of the feature; and determining said calibration information in the form of a focal length and/or a field of view of the digital camera based on said current distance and said physical limbus size dimension. ### Claim 2 (depends on 1) Method for detecting a current gaze point of a second user on an entity, the method comprising: determining calibration information with respect to a digital camera, the determining of calibration information being in accordance with claim 1; receiving from the digital camera at least one tracking image, being a depiction of at least one eye of the second user; performing digital image processing on the tracking image to locate and determine a facial measure, of a facial feature of the second user, the facial feature being visible in said tracking image; and detecting said current gaze point of the second user based on a detected gaze direction of the second user, said facial measure, said calibration information and a spatial interrelationship between the digital camera and the entity. ### Claim 3 (depends on 2) Method according to claim 2, further comprising: determining a current distance between the eye of the second user and the entity based on said facial measure, said calibration information and a spatial interrelationship between the digital camera and the entity; and wherein the current gaze point of the second user is determined based on said detected gaze direction of the second user and said current distance between the eye of the second user and the entity. ### Claim 4 (depends on 2) Method according to claim 2, wherein said facial measure is selected from a group: a corneal reflection, a face landmark, an inter-pupil distance, a limbus size, and a sclera size. ### Claim 5 (depends on 2) Method according to claim 2, wherein said in-image feature size dimension and said in-image limbus size dimension are both measured in terms of a number of pixels of the calibration image. ### Claim 6 (depends on 5) Method according to claim 5, wherein said digital image processing performed on the calibration image comprises an edge detection step or feature detection step using interpolation between intensity measures of individual adjacent pixels in the calibration image so as to detect the in-image feature size dimension and/or said in-image limbus size dimension with sub-pixel precision. ### Claim 7 (depends on 2) Method according to claim 2, wherein said physical limbus size dimension is a physical distance, such as expressed in mm. ### Claim 8 (depends on 2) Method according to claim 2, wherein the feature covers at least 50% of the entity and has high contrast in relation to other parts of the entity, or the feature covers 100% of the entity and has high brightness. ### Claim 9 (depends on 2) Method according to claim 2, wherein the feature has a rectangular shape with straight, sharp edges, or is in the form of a known coarse pattern comprising between 2 and 10 fields that in turn have straight, sharp edges. ### Claim 10 (depends on 2) Method according to claim 2, wherein the entity is a screen of an electronic display. ### Claim 11 (depends on 10) Method according to claim 10, wherein the feature is deliberately displayed on the screen as part of a camera calibration information determining step, or the feature is constituted by the screen showing a generally bright full screen background with only a small number of pixels constituting a foreground; the method comprises controlling the digital camera to depict said at least one eye repeatedly to produce a series of calibration images; and the feature automatically being detected, using digital image processing, as a present, generally rectangular, bright reflected shape in said eye in at least one of the hence produced calibration images. ### Claim 12 (depends on 11) Method according to claim 11, wherein the method further comprises using a trained neural network, trained using a set of possible screen contents occurring during use of at least one computer, to perform said automatic detection of the feature. ### Claim 13 (depends on 2) Method according to claim 2, wherein the method further comprises using a trained neural network to determine an expected limbus size dimension based on the calibration image, the calibration image depicting the face of the first user; and using the expected limbus size dimension as said physical limbus size dimension. ### Claim 14 (depends on 2) Method according to claim 2, wherein a distance between the eye of the first user and the digital camera is between 30 and 80 cm when the calibration image is produced. ### Claim 15 (depends on 2) Method according to claim 2, wherein a focal plane of the digital camera is arranged in a main viewing plane of the entity±5 cm perpendicularly to said main viewing plane. ### Claim 16 System for determining a gaze point of a second user in relation to an entity using a digital camera, the system comprising a calibration module, arranged to determine calibration information in relation to the digital camera by receiving from the digital camera at least one calibration image, being a depiction of at least one eye of a first user, wherein the first user may be the same as or different from the second user; performing digital image processing on the calibration image to locate and determine an in-image feature size dimension of a feature of the entity as reflected in said eye of the first user; and to locate a limbus of said eye of the first user and determine an in-image limbus size dimension of said limbus; determining a current distance between the eye of the first user and the entity based on said in-image feature size dimension, said in-image limbus size dimension, a curvature of the cornea of the eye of the first user, a physical limbus size dimension of the eye of the first user and a physical size of the feature; and determine said calibration information in the form of a focal length and/or a field of view of the digital camera based on said current distance and said physical limbus size dimension, the system further comprising a gaze point detection module, arranged to detect a current gaze point of a second user, the second user being the first user or a different user, by receiving from the digital camera a digital tracking image, depicting at least one eye of the second user; performing digital image processing on the tracking image to locate and determine a facial measure, of a facial feature of the second user, the facial feature being visible in said tracking image; and detecting said current gaze point of the second user based on a detected gaze direction of the second user, said facial measure, said calibration information and a spatial interrelationship between the digital camera and the entity. ### Claim 17 (depends on 2) A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of a method according claim 2.