# US 2020/0401219 A1 · Eye tracking calibration for a surgical robotic system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401219 A1 - Document ID: US20200401219A1 - Status: Published application - Kind: A1 - Application number: 16/449,170 - Filed: 2019-06-21 - Published: 2020-12-24 - Earliest claimed: 2019-06-21 - Assignees: Verb Surgical Inc. - Applicants: Verb Surgical Inc. - Inventors: Anette Lia Freiin von Kapri; Denise Miller; Joan Savall - Web page: https://patentgrep.com/patent/US20200401219A1 ## CPC - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - A61B 34/25: {User interfaces for surgical systems} - A61B 34/30: Surgical robots - A61B 90/361: {Image-producing devices, e.g. surgical cameras} - A61B 2017/00216: {with eye tracking or head position tracking control} - A61B 2017/00725: {Calibration or performance testing} - A61B 2090/371: {with simultaneous use of two cameras} - G06V 40/19: Sensors therefor - G06V 40/193: {Preprocessing; Feature extraction} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method for passively calibrating and verifying eye tracking in a surgical robotic system. The gaze of a user facing a user display of a user console of a surgical robotic system is tracked while the user is using a user interface device (UID). When a user interaction to the UID is detected, an expected gaze position on the display of the user console is determined without instructing the user to look at the expected gaze position. The latter becomes a reference gaze point of the user at the time of detected user action. The measured gaze point of the user is compared with an acceptable threshold of the reference gaze point, and in response to a determination of a mismatch, calibration parameters used by the tracking are adjusted according to the reference gaze point. Other aspects are also described and claimed. ## Claims ### Claim 1 A method for eye tracking in a surgical robotic system, comprising: tracking a gaze of a user facing a display of a user console of a surgical robotic system while the user is using a user interface device (UID) thereby producing a measured gaze point of the user; detecting that the user is interacting with the UID thereby producing a detected user interaction; determining an expected gaze point on the display of the user console as a reference gaze point of the user at the time of the detected user interaction; determining whether the measured gaze point is within an acceptable threshold of the reference gaze point at the time of the detected user interaction; and in response to determining the measured gaze point is within the acceptable threshold of the reference gaze point, continuing said tracking the gaze of the user without performing an eye tracking calibration. ### Claim 2 (depends on 1) The method of claim 1, wherein tracking the gaze of the user comprises utilizing an eye tracking sensor to calculate geometry and gaze point of an eye of the user. ### Claim 3 (depends on 1) The method of claim 1, wherein the detected user interaction comprises user login. ### Claim 4 (depends on 1) The method of claim 1, wherein the detected user interaction comprises interacting with graphical user interface, GUI, elements on the user display. ### Claim 5 (depends on 1) The method of claim 1, wherein the user display is showing an endoscopic view of a surgical site and the detected user interaction comprises manipulation of the UID by the user which causes actuation of an end effector within the surgical site, and wherein the expected gaze point is a determined position of the end effector on the user display. ### Claim 6 (depends on 1) The method of claim 1, wherein said tracking the gaze of the user comprises performing an eye tracking algorithm that uses a plurality of parameters that have been adjusted by performing the eye tracking calibration method, and wherein in response to determining the measured gaze point is within the acceptable threshold, continuing said tracking without adjusting the plurality of parameters. ### Claim 7 (depends on 1) The method of claim 1, wherein in response to determining that the measured gaze point is within a first acceptable threshold of the reference gaze point, the one or more parameters used for tracking the gaze of the user are adjusted until the tracking produces the reference gaze point. ### Claim 8 (depends on 7) The method of claim 7 wherein in response to determining that the measured gaze point is outside the first acceptable threshold but inside a second acceptable threshold of the reference gaze point, the method further comprises: selecting a stored profile from a database of stored profiles, wherein the stored profile contains a plurality of parameters to be used by an eye tracking algorithm; and re-configuring an eye tracking algorithm that is for tracking the gaze of the user, in accordance with the plurality of parameters in the selected stored profile. ### Claim 9 (depends on 8) The method of claim 8 further comprising one of: a) in response to determining that the measured gaze point as produced by the reconfigured tracking algorithm is a better match to the reference gaze point, continuing the tracking in accordance with the selected stored profile; or b) in response to determining that the measured gaze point as produced by the reconfigured tracking algorithm is not a better match to the reference gaze point, performing an eye tracking calibration process to create a new profile. ### Claim 10 A method for calibrating eye tracking in a surgical robotic system, comprising: generating a measured gaze by tracking a gaze of a user facing a user display of a user console of a surgical robotic system while the user is using a user interface device (UID); detecting a user action to the UID; determining an expected gaze position on the display of the user console as a reference gaze of the user at the time of detected user action; determining whether a mismatch has occurred, wherein a mismatch has occurred if the tracked gaze of the user is outside a first threshold and within a second threshold of the reference gaze at the time of detected user action; and in response to a determination of a mismatch, adjusting one or more parameters used by the eye tracking of the user gaze until the eye tracking produces the reference gaze. ### Claim 11 (depends on 10) The method of claim 10, further comprising determining if the difference between the reference gaze and the measured gaze is outside the second threshold, and if so then selecting a different profile to be used by the eye tracking to generate a new measured gaze, and then determining if the new measured gaze is a closer match to the reference gaze, and if so, selecting the different profile for use by the eye tracking. ### Claim 12 (depends on 10) The method of claim 10, further comprising determining if the difference between the reference gaze and the measured gaze is outside the second threshold, and if so then selecting a different profile to be used by the eye tracking to generate a new measured gaze, and then determining if the new measured gaze is a closer match to the reference gaze, and if not, performing an eye tracking calibration method to produce a new profile. ### Claim 13 (depends on 10) The method of claim 10 wherein determining an expected gaze position on the display of the user console as a reference gaze of the user at the time of detected user action comprises: identifying an on-screen position of a visual element that is not generated as part of an eye tracking calibration process; determining how the measured gaze changes over time, before and after the visual element appears on screen; and determining vicinity of the measured gaze to the visual element on screen. ### Claim 14 (depends on 13) The method of claim 13 wherein the detected user action to the UID is interaction of the user with the visual element being a mouse cursor or a dialog box. ### Claim 15 A device for calibrating eye tracking in a surgical robotic system, comprising: a camera aimed at a user seat, wherein the user seat is facing a user display of a user console of a surgical robotic system having a user interface device (UID) that is to be manipulated by a user; a memory having instructions stored therein; and one or more processors to execute the instructions to detect a user action by a user to the user console or to the UID, based on the detected user action, determine an expected gaze position on the user display of the user console as a reference gaze of the user at the time of detected user action, track a gaze of the user using an eye tracking algorithm that receives as input image data from the camera and uses a plurality of eye tracking calibration parameters, determine whether the tracked gaze of the user is within an acceptable threshold of the reference gaze at the time of detected user action, and continuing to track the gaze of the user without changing the eye tracking calibration parameters, in response to a determination that the tracked gaze is within the acceptable threshold. ### Claim 16 (depends on 15) The device of claim 15, wherein the detected user action is user login. ### Claim 17 (depends on 15) The device of claim 15, wherein the detected user action is interaction with GUI elements on the user display. ### Claim 18 (depends on 15) The device of claim 15, wherein the user display is showing an endoscopic view of a surgical site and the detected user interaction comprises manipulation of the UID by the user, wherein the manipulation of the UID causes actuation of an end effector within the surgical site, and wherein the expected gaze position is a determined position of the end effector on the user display. ### Claim 19 (depends on 18) The device of claim 18 wherein the processor is to execute the instructions to determine the position of the end effector on the user display by: determining a joint state of a robotic arm to which the end effector is attached and is being controlled in accordance with manipulation by the user of the UID; transforming the joint state of the robotic arm to endoscope image frame domain; transforming from the endoscope image frame domain to user display frame domain.