# US 12,758,340 B2 · Systems and methods for remote mentoring in a robot assisted medical system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,758,340 B2 - Document ID: US12758340B2 - Status: Granted - Kind: B2 - Application number: 18/835,266 - Filed: 2023-01-25 - Published: 2025-05-01 - Granted: 2026-10-13 - Earliest claimed: 2022-02-02 - Assignees: Intuitive Surgical Operations, Inc. - Applicants: Intuitive Surgical Operations, Inc. - Inventors: Sundar Murugappan; Mitchell Doughty; Govinda Payyavula; Matthew S. Warren - Web page: https://patentgrep.com/patent/US12758340B2 ## CPC - A61B 34/37: Leader-follower robots (A61B34/35 takes precedence) - A61B 34/25: {User interfaces for surgical systems} - A61B 90/361: {Image-producing devices, e.g. surgical cameras} - A61B 2090/3614: {using optical fibre} - A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image} - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) ## Abstract A non-transitory machine-readable media may store instructions that, when run by one or more processors, cause the one or more processors to generate stereo endoscopic image data of a medical field and define a medical field volume from the stereo endoscopic image data. The processors may also project a 3D representation of the medical field volume to an extended-reality display device in a remote volume. The 3D representation may include a stereoscopic image and a 3D scene generated from the stereo endoscopic image data. The processors may also generate visual guidance in the remote volume to augment the stereo endoscopic image, map the visual guidance from the remote volume to the medical field volume to generate an augmented image of the medical field volume, and project the augmented image of the medical field volume to a display device viewed by an operator of instruments in the medical field. ## Claims ### Claim 1 A non-transitory machine-readable media storing instructions that, when run by one or more processors, cause the one or more processors to: generate stereo endoscopic image data of a medical field; define a medical field volume from the stereo endoscopic image data; project a three dimensional representation of the medical field volume to an extended-reality display device in a remote volume, the three dimensional representation including a stereoscopic image generated from the stereo endoscopic image data and a three-dimensional scene representing the medical field; generate visual guidance in the remote volume to augment the stereo endoscopic image, wherein the visual guidance is generated from a user's physical expressions in the remote volume; map the visual guidance from the remote volume to the medical field volume to generate an augmented image of the medical field volume; and project the augmented image of the medical field volume to a first display device viewed by an operator of one or more instruments in the medical field. ### Claim 2 (depends on 1) The non-transitory machine-readable media of claim 1, wherein the stereo endoscopic image data is generated by an endoscopic imaging device controlled by a robot-assisted manipulator. ### Claim 3 (depends on 2) The non-transitory machine-readable media of claim 2, wherein the medical field includes tools controlled by the robot-assisted manipulator. ### Claim 4 (depends on 1) The non-transitory machine-readable media of claim 1, wherein the medical field volume includes patient anatomy and instruments visible stereo endoscopic image data of the medical field. ### Claim 5 (depends on 1) The non-transitory machine-readable media of claim 1, wherein the medical field volume includes user interface elements. ### Claim 6 (depends on 1) The non-transitory machine-readable media of claim 1 storing instructions that, when run by one or more processors, further cause the one or more processors to receive audio input from an operator at an operator console. ### Claim 7 (depends on 1) The non-transitory machine-readable media of claim 1, wherein the three-dimensional scene is generated by determining an object depth for an object in the stereoscopic image. ### Claim 8 (depends on 1) The non-transitory machine-readable media of claim 1, wherein the three-dimensional scene is generated for a portion of the stereoscopic image. ### Claim 9 (depends on 1) The non-transitory machine-readable media of claim 1, wherein the remote volume moves with extended-reality display device. ### Claim 10 (depends on 1) The non-transitory machine-readable media of claim 1 storing instructions that, when run by one or more processors, further cause the one or more processors to, receive sensor information from the extended-reality display device, wherein the sensor information tracks the user's physical expressions in the remote volume. ### Claim 11 (depends on 1) The non-transitory machine-readable media of claim 1, wherein the user's physical expressions in the remote volume include at least one of freehand gestures, eye gaze, or head motion of a user in the remote volume. ### Claim 12 (depends on 1) The non-transitory machine-readable media of claim 1, wherein the visual guidance includes at least one of include an arrow, a pointer, a ghost hand, a ghost tool, a gaze marker, a virtual anatomical model, or annotations. ### Claim 13 (depends on 1) The non-transitory machine-readable media of claim 1 storing instructions that, when run by one or more processors, further cause the one or more processors to generate audio guidance. ### Claim 14 (depends on 1) The non-transitory machine-readable media of claim 1, wherein mapping the visual guidance includes determining a three-dimensional position for the visual guidance in the medical field volume. ### Claim 15 (depends on 1) The non-transitory machine-readable media of claim 1, wherein mapping the visual guidance includes scaling a parameter from the remote volume to the medical field volume. ### Claim 16 (depends on 1) The non-transitory machine-readable media of claim 1 storing instructions that, when run by one or more processors, further cause the one or more processors to adaptively change the mapping as an endoscopic imaging system moves. ### Claim 17 (depends on 1) The non-transitory machine-readable media of claim 1 storing instructions that, when run by one or more processors, further cause the one or more processors to project a model of a manipulator assembly from the extended-reality display device. ### Claim 18 (depends on 17) The non-transitory machine-readable media of claim 17 storing instructions that, when run by one or more processors, further cause the one or more processors to fuse the model of the manipulator assembly with the augmented image of the medical field volume for display on the extended-reality display device. ### Claim 19 (depends on 1) The non-transitory machine-readable media of claim 1 storing instructions that, when run by one or more processors, further cause the one or more processors to project a model of an operator console from the extended-reality display device. ### Claim 20 (depends on 1) The non-transitory machine-readable media of claim 1, wherein the medical field volume may be a mosaic comprised of a plurality of images and the visual guidance may be visible in an area of the medical field volume visible by moving an endoscopic imaging instrument.