Systems and methods of communicating through visual overlay for surgical medical systems

A surgical system comprising:

• a master controller for controlling one or more surgical tools;
• a viewer on the master controller configured to display a representation of a surgical site;
• a screen separate from the master controller configured to display the representation of the surgical site;
• and
• an input on the master controller configured to change the master controller from a teleoperation mode into a virtual marking mode,
• wherein, in the teleoperation mode, the master controller is configured to move or actuate the one or more surgical tools, and, in the virtual marking mode, the master controller is configured to generate a virtual marker overlayed on the representation of the surgical site in the screen separate from the master controller,
• wherein in the virtual marking mode, the screen separate from the master controller is operable to one or both of (i) modify the virtual marker generated at the master controller and overlayed on the representation of the surgical site to thereby provide a modified virtual marker, or (ii) generate another virtual marker overlayed on the representation of the surgical site to thereby provide a generated other virtual marker,
• wherein the viewer on the master controller is further configured to display one or both of (i) the modified virtual marker, or (ii) the generated other virtual marker.

A method of communication during surgery comprising:

• displaying, at a viewer of a master controller, a representation of a surgical site;
• displaying, at a screen separate from the master controller, the representation of the surgical site;
• controlling, in a teleoperation mode, one or more surgical tools in response to user input at the master controller;
• changing the master controller from the teleoperation mode into a virtual marking mode in response to user input at the master controller;
• generating, in the virtual marking mode, a virtual marker overlayed on the representation of the surgical site at the screen separate from the master controller in response to user input at the master controller, the virtual marker being defined by a user by manipulating the user input at the master controller such that the virtual marker is a user-defined virtual marker, the user-defined virtual marker being defined in relation to an anatomical structure at the surgical site to thereby associate the user-defined virtual marker with the anatomical structure;
• and
• fixing the user-defined virtual marker to a point in the representation of the surgical site, the point positioned at the anatomical structure associated with the user-defined virtual marker, wherein the user-defined virtual marker remains fixed to the point in the representation of the surgical site when a camera view changes.