# US 10,871,823 B1 · Systems and methods for using scene understanding for calibrating eye tracking > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,871,823 B1 - Document ID: US10871823B1 - Status: Granted - Kind: B1 - Application number: 16/420,983 - Filed: 2019-05-23 - Granted: 2020-12-22 - Earliest claimed: 2019-05-23 - Assignees: Facebook Technologies, LLC - Applicants: Facebook Technologies, LLC - Inventors: Kirk Erik Burgess; Sebastian Sztuk - Web page: https://patentgrep.com/patent/US10871823B1 ## CPC - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30041: Eye; Retina; Ophthalmic - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A system can include one or more processors that determine a first position of an object in view of a user of a head mounted display (HMD) and a first gaze direction towards the first position. The object can be detected by one or more sensors. The one or more processors detect that the user is gazing at the object. The one or more processors determine, responsive to detecting that the user is gazing at the object, a second gaze direction based on at least a second position of one or more eyes of the user of the HMD provided via an eye tracking operation of the HMD. The one or more processors calibrate, based at least on a difference between the first gaze direction and the second gaze direction, the eye tracking operation of the HMD. ## Claims ### Claim 1 A method for calibrating eye tracking operation, the method comprising: determining, by one or more processors of a head mounted display (HMD), a first position of an object in view of a user of the HMD and a first predicted gaze direction towards the first position using the first position of the object, the object detected by one or more sensors; detecting, by the one or more processors, that the user is gazing at the object; determining, by the one or more processors responsive to detecting that the user is gazing at the object, a second gaze direction based on at least a second position of one or more eyes of the user of the HMD provided via an eye tracking operation of the HMD; determining, by the one or more processors, a first vergence plane of the first predicted gaze direction and a second vergence plane of the second gaze direction; and calibrating, by the one or more processors, based at least on a difference between the first vergence plane of the first predicted gaze direction and the second vergence plane of the second gaze direction, the eye tracking operation of the HMD. ### Claim 2 (depends on 1) The method of claim 1, further comprising prompting, by the one or more processors, the user of the HMD to gaze at the detected object. ### Claim 3 (depends on 1) The method of claim 1, further comprising determining the first position of the object responsive to detecting, by the one or more processors, one of the object coming into view of the one or more sensors or the object in a first image captured subsequent to a second image in which the object is not detected. ### Claim 4 (depends on 1) The method of claim 1, further comprising detecting, by the one or more sensors, that the user is gazing at the object responsive to an interaction of the user with the object. ### Claim 5 (depends on 1) The method of claim 1, further comprising detecting, by the one or more sensors, that the user is gazing at the object responsive to the user reading text associated with the object. ### Claim 6 (depends on 1) The method of claim 1, further comprising: detecting, by the one or more processors during use of the HMD, a plurality of objects and a plurality of gaze directions corresponding to the plurality of objects; and calibrating, by the one or more processors, the eye tracking operation responsive to each of a plurality of differences between the plurality of gaze directions and corresponding gaze directions provided via the eye tracking operation. ### Claim 7 A system, comprising: one or more processors configured to: determine a first position of an object in view of a user of the HMD and a first predicted gaze direction towards the first position using the first position of the object, the object detected by one or more sensors; detect that the user is gazing at the object; determine, responsive to detecting that the user is gazing at the object, a second gaze direction based on at least a second position of one or more eyes of the user of the HMD provided via an eye tracking operation of the HMD; determine a first vergence plane of the first predicted gaze direction and a second vergence plane of the second gaze direction, and calibrate, based at least on a difference between the first vergence plane of the first predicted gaze direction and the second vergence plane of the second gaze direction, the eye tracking operation of the HMD. ### Claim 8 (depends on 7) The system of claim 7, wherein the one or more processors are further configured to prompt the user of the HMD to gaze at the detected object. ### Claim 9 (depends on 7) The system of claim 7, wherein the one or more processors are further configured to determine the first position of the object responsive to detecting one of the object coming into view of the one or more sensors or the object in a first image captured subsequent to a second image in which the object is not detected. ### Claim 10 (depends on 7) The system of claim 7, wherein the one or more processors are further configured to detect that the user is gazing at the object responsive to an interaction of the user with the object. ### Claim 11 (depends on 7) The system of claim 7, wherein the one or more processors are further configured to detect that the user is gazing at the object responsive to the user reading text associated with the object. ### Claim 12 (depends on 7) The system of claim 7, wherein the one or more processors are further configured to: detect, during use of the HMD, a plurality of objects and a plurality of gaze directions corresponding to the plurality of objects; and calibrate the eye tracking operation responsive to each of a plurality of differences between the plurality of gaze directions and corresponding gaze directions provided via the eye tracking operation. ### Claim 13 A computer-readable medium comprising processor-executable instructions that when executed by one or more processors, cause the one or more processors to: determine a first position of an object in view of a user of the HMD and a first predicted gaze direction towards the first position using the first position of the object, the object detected by one or more sensors; detect that the user is gazing at the object; determine, responsive to detecting that the user is gazing at the object, a second gaze direction based on at least a second position of one or more eyes of the user of the HMD provided via an eye tracking operation of the HMD; determine a first vergence plane of the first predicted gaze direction and a second vergence plane of the second gaze direction; and calibrate, based at least on a difference between the first vergence plane of the first predicted gaze direction and the second vergence plane of the second gaze direction, the eye tracking operation of the HMD. ### Claim 14 (depends on 13) The computer-readable medium of claim 13, further comprising instructions that cause the one or more processors to prompt the user of the HMD to gaze at the detected object. ### Claim 15 (depends on 13) The computer-readable medium of claim 13, further comprising instructions that cause the one or more processors to determine the first position of the object responsive to detecting one of the object coming into view of the one or more sensors or the object in a first image captured subsequent to a second image in which the object is not detected. ### Claim 16 (depends on 13) The computer-readable medium of claim 13, further comprising instructions that cause the one or more processors to detect that the user is gazing at the object responsive to detecting at least one of an interaction of the user with the object or the user reading text associated with the object. ### Claim 17 (depends on 13) The computer-readable medium of claim 13, further comprising instructions that cause the one or more processors to: detect, during use of the HMD, a plurality of objects and a plurality of gaze directions corresponding to the plurality of objects; and calibrate the eye tracking operation responsive to each of a plurality of differences between the plurality of gaze directions and corresponding gaze directions provided via the eye tracking operation.