# US 11,998,291 B2 · Robotic surgical systems with user engagement monitoring > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,998,291 B2 - Document ID: US11998291B2 - Status: Granted - Kind: B2 - Application number: 16/966,666 - Filed: 2019-01-02 - Published: 2021-02-04 - Granted: 2024-06-04 - Earliest claimed: 2018-02-02 - Assignees: Covidien LP - Applicants: Covidien LP - Inventors: William J. Peine - Web page: https://patentgrep.com/patent/US11998291B2 ## CPC - A61B 34/35: for telesurgery - A61B 34/37: Leader-follower robots (A61B34/35 takes precedence) - A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis - A61B 34/74: {Manipulators with manual electric input means} - A61B 34/76: {Manipulators having means for providing feel, e.g. force or tactile feedback} - A61B 34/77: {Manipulators with motion or force scaling} - A61B 90/03: {Automatic limiting or abutting means, e.g. for safety} - A61B 90/37: {Surgical systems with images on a monitor during operation} - H04N 13/302: for viewing without the aid of special glasses, i.e. using autostereoscopic displays - A61B 2017/00216: {with eye tracking or head position tracking control} - A61B 2034/2055: {Optical tracking systems} - A61B 2034/2065: {Tracking using image or pattern recognition} - A61B 2034/742: {Joysticks} - A61B 2090/371: {with simultaneous use of two cameras} - A61B 2090/502: {Headgear, e.g. helmet, spectacles} - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A robotic surgical system with user engagement monitoring includes a robot assembly, a surgeon console, and a tracking device. The robot assembly includes a robotic arm coupled to a surgical instrument. The surgeon console includes a display device and a handle communicatively coupled to at least one of the robot assembly, the robotic arm, or the surgical instrument. The tracking device includes an image capture device configured to capture an image of a user position reference point. At least one of the surgeon console or the tracking device is configured to compute a position of the user position reference point based on the image; determine whether a user is engaged with the surgeon console based on the computed position; and, in response to a determination that the user is disengaged from the surgeon console, cause the robotic surgical system to operate in a safe mode. ## Claims ### Claim 1 A robotic surgical system with user engagement monitoring, comprising: a robot assembly including a robotic arm coupled to a surgical instrument; a surgeon console including: a handle communicatively coupled to at least one of the robot assembly, the robotic arm, or the surgical instrument, and a display device; and a tracking device including an image capture device configured to capture an image of a user position reference point, wherein at least one of the surgeon console or the tracking device is configured to: compute, based on the captured image, a position of the user position reference point relative to the display device, determine whether a user is engaged with or disengaged from the surgeon console based on the computed position, and in response to a determination that the user is disengaged from the surgeon console, cause the robotic surgical system to operate in a safe mode, wherein at least one of the surgeon console or the tracking device is further configured to compute the position of the user position reference point by generating location data corresponding to at least one of the position, or an orientation, of the user position reference point, within a three-dimensional coordinate space, relative to the display device; wherein, in the determination of whether the user is engaged with or disengaged from the surgeon console, at least one of the surgeon console or the tracking device is further configured to: compute a difference angle based on the position and orientation of the user position reference point relative to the display device; compare the difference angle to a first threshold angle; in response to a determination that the difference angle is greater than the first threshold angle, determine that the user is disengaged from the surgeon console; compute, based on the position and the orientation of the user position reference point, a direction of movement of the user position reference point relative to the display device; and select the first threshold angle based on the direction of movement of the user position reference point for use in commanding movement of the robot assembly. ### Claim 2 (depends on 1) The robotic surgical system of claim 1, wherein at least one of the surgeon console or the tracking device is further configured to select the first threshold angle from a plurality of first threshold angles based on the position and the orientation of the user position reference point relative to the display device. ### Claim 3 (depends on 1) The robotic surgical system of claim 1, wherein, in the determination of whether the user is engaged with or disengaged from the surgeon console, at least one of the surgeon console or the tracking device is further configured to: in response to a determination that the difference angle is less than the first threshold angle, determine whether the difference angle is less than a second threshold angle that is smaller than the first threshold angle; and in response to a determination that the difference angle is less than the second threshold angle, determine that the user is engaged with the surgeon console. ### Claim 4 (depends on 3) The robotic surgical system of claim 3, wherein at least one of the surgeon console or the tracking device is further configured to, in response to the determination that the user is engaged with the surgeon console, cause the robotic surgical system to exit the safe mode. ### Claim 5 (depends on 1) The robotic surgical system of claim 1, wherein at least one of the surgeon console or the tracking device is further configured to, at a time when the robotic surgical system operates in the safe mode: in response to a determination that the user is engaged with the surgeon console, cause the robotic surgical system to exit the safe mode after an elapsing of a threshold amount of time after the determination that the user is engaged. ### Claim 6 A robotic surgical system with user engagement monitoring, comprising: a robot assembly including a robotic arm coupled to a surgical instrument; a surgeon console including: a handle communicatively coupled to at least one of the robot assembly, the robotic arm, or the surgical instrument, and a display device; and a tracking device including an image capture device configured to capture an image of a user position reference point, wherein at least one of the surgeon console or the tracking device is configured to: compute, based on the captured image, a position of the user position reference point relative to the display device, determine whether a user is engaged with or disengaged from the surgeon console based on the computed position, and in response to a determination that the user is disengaged from the surgeon console, cause the robotic surgical system to operate in a safe mode; at a time when the robotic surgical system operates in the safe mode: restrict movement of the handle from a previous position of the handle, and transmit, to the computing device, instructions to restrict movement of at least one of the robot assembly, the robotic arm, or the surgical instrument; wherein the computing device is configured to: receive the instructions, and transmit the instructions to at least one of the robot assembly, the robotic arm, or the surgical instrument; and wherein at least one of the robotic arm, the robot assembly, or the surgical instrument is configured to: receive the instructions, and restrict movement of at least one of the robot assembly, the robotic arm, or the surgical instrument in response to the instructions. ### Claim 7 (depends on 1) The robotic surgical system of claim 1, wherein at least one of the surgeon console or the tracking device is further configured to, at a time when the robotic surgical system operates in the safe mode: prevent a movement of the handle from causing a corresponding movement of the robotic arm communicatively coupled to the handle. ### Claim 8 A robotic surgical system with user engagement monitoring, comprising: a robot assembly including a robotic arm coupled to a surgical instrument; a surgeon console including: a handle communicatively coupled to at least one of the robot assembly, the robotic arm, or the surgical instrument, and a display device; and a tracking device including an image capture device configured to capture an image of a user position reference point, wherein at least one of the surgeon console or the tracking device is configured to: compute, based on the captured image, a position of the user position reference point relative to the display device, determine whether a user is engaged with or disengaged from the surgeon console based on the computed position, in response to a determination that the user is disengaged from the surgeon console, cause the robotic surgical system to operate in a safe mode; detect an amount of movement of the handle; determine, based on the amount of movement of the handle, an amount of movement of at least one of the robot assembly, the robotic arm, or the surgical instrument to be caused in response to the movement of the handle; cause at least one of the robot assembly, the robotic arm, or the surgical instrument to move by the determined amount of movement, wherein, at a time when the robotic surgical system operates in the safe mode, the determination of the amount of movement of at least one of the robot assembly, the robotic arm, or the surgical instrument to be caused includes applying a downward scaling factor. ### Claim 9 (depends on 8) The robotic surgical system of claim 8, wherein at least one of the surgeon console or the tracking device is further configured to: compute a velocity of a movement of the handle; and modify the downward scaling factor based on the velocity. ### Claim 10 (depends on 1) The robotic surgical system of claim 1, wherein the surgeon console includes a plurality of motors corresponding to the handle, each of the motors being operably coupled to the handle and being associated with a direction of movement of the handle, wherein, at a time when the robotic surgical system operates in the safe mode, at least one of the surgeon console or the tracking device is further configured to: compute a velocity of a movement of the handle, compute a direction of the movement of the handle, compute, based on the velocity of the movement of the handle, a force in a direction opposite to the direction of the movement of the handle, identify, among the plurality of motors of the handle, a motor associated with the direction opposite to the direction of the movement of the handle, and cause actuation of the identified motor in the direction opposite to the direction of the movement of the handle to generate the computed force in the direction opposite to the direction of the movement of the handle. ### Claim 11 A robotic surgical system with user engagement monitoring, comprising: a robot assembly including a robotic arm coupled to a surgical instrument; a surgeon console including: a handle communicatively coupled to at least one of the robot assembly, the robotic arm, or the surgical instrument, a plurality of motors, wherein the plurality of motors are operably coupled to the handle and are associated with a plurality of directions, respectively, of movement of the handle, and a display device; and a tracking device including an image capture device configured to capture an image of a user position reference point, wherein at least one of the surgeon console or the tracking device is configured to: compute, based on the captured image, a position of the user position reference point relative to the display device, determine whether a user is engaged with or disengaged from the surgeon console based on the computed position, in response to the determination that the user is disengaged with the surgeon console: cause the robotic surgical system to operate in a safe mode, identify a first position of the handle, compute a distance traveled by the handle from the first position of the handle, compute a direction of the movement of the handle, compute, based on the distance, a force in a direction opposite to the direction of the movement of the handle, identify, among the plurality of motors of the handle, a motor associated with the direction opposite to the direction of the movement of the handle, and cause actuation of the identified motor in the direction opposite to the direction of the movement of the handle to generate the computed force in the direction opposite to the direction of the movement of the handle, wherein the surgeon console is further configured to: actuate the motor in the direction opposite to the direction of the movement of the handle until the handle is positioned in the first position. ### Claim 12 (depends on 1) The robotic surgical system of claim 1, further comprising eyewear including a plurality of markers, wherein the user position reference point includes at least one of the plurality of markers. ### Claim 13 (depends on 1) The robotic surgical system of claim 1, wherein the user position reference point includes at least one of an eye, a head, or another portion of the user. ### Claim 14 (depends on 1) The robotic surgical system of claim 1, wherein the display device is an autostereoscopic display device.