# US 2022/0031406 A1 · Method of graphically tagging and recalling identified structures under visualization for robotic surgery

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0031406 A1
- Document ID: US20220031406A1
- Status: Published application
- Kind: A1
- Application number: 17/499,822
- Filed: 2021-10-12
- Published: 2022-02-03
- Earliest claimed: 2017-06-23
- Assignees: Asensus Surgical US, Inc.
- Applicants: Asensus Surgical US, Inc.
- Inventors: Kevin Andrew Hufford; David J. Meagher
- Web page: https://patentgrep.com/patent/US20220031406A1

## CPC

- A61B 90/361: {Image-producing devices, e.g. surgical cameras}
- A61B 1/000094: {extracting biological structures}
- A61B 1/0005: {combining images e.g. side-by-side, superimposed or tiled}
- A61B 1/00055: {for alerting the user}
- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- A61B 34/25: {User interfaces for surgical systems}
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/0016: {involving temporal comparison}
- G06T 7/40: Analysis of texture (depth or shape recovery from texture G06T7/529)
- G06T 7/50: Depth or shape recovery
- G06T 7/90: Determination of colour characteristics
- G06T 11/00: Two-dimensional [2D] image generation
- A61B 2017/00203: {with speech control or speech recognition}
- A61B 2017/00216: {with eye tracking or head position tracking control}
- A61B 2017/00973: {pedal-operated}
- A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones}
- A61B 2034/107: {Visualisation of planned trajectories or target regions}
- A61B 2034/2065: {Tracking using image or pattern recognition}
- A61B 2034/254: {being adapted depending on the stage of the surgical procedure}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/10068: Endoscopic image
- G06T 2207/30101: Blood vessel; Artery; Vein; Vascular
- G06T 2207/30168: Image quality inspection
- G06T 2210/41: Medical
- G06T 2210/62: Semi-transparency
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A system and method for augmenting an endoscopic display during a medical procedure including capturing a real-time image of a working space within a body cavity during a medical procedure. A feature of interest in the image is identified to the system using a user input handle of a surgical robotic system, and a graphical tag is displayed on the image marking the feature.

## Claims

### Claim 1

A method of tagging regions of interest on displayed images during a medical procedure, comprising:
positioning an endoscope in a body cavity;
positioning a surgical instrument in the body cavity;
mounting the surgical instrument to a first robotic manipulator arm;
causing the first robotic manipulator arm to manipulate the surgical instrument within the body cavity in in response to user manipulation of a user input device;
capturing images of a surgical site within the body cavity and displaying the images on a display, the displayed images including images of the surgical instrument,
entering a tag selection mode comprising:
displaying a graphical pointer as an overlay on the display;
in response to user manipulation of the user input, positioning the graphical pointer at a region of interest on images of the surgical site displayed on the display;
in response to user selection input, displaying a graphical tag at the location of the graphical pointer on the images displayed on the display.

### Claim 2 (depends on 1)

The method of claim 1, wherein the method further comprises:
exiting the tag selection mode, wherein graphical tag remains displayed at the region of interest on images of the surgical site displayed on the display.

### Claim 3 (depends on 1)

The method of claim 1, wherein the method further includes removing the graphical tag in response to user input to remove the tag.

### Claim 4 (depends on 3)

The method of claim 3, wherein the method further includes restoring the graphical tag at the region of interest in response to user input to restore the tag.

### Claim 5 (depends on 1)

The method of claim 1, wherein the user input includes a user input handle, and wherein
the causing step causes the first robotic manipulator arm to manipulate the surgical instrument in response to user manipulation of the user input handle,
the step of positioning the graphical pointer at a region of interest is in response to user manipulation of the user input handle.

### Claim 6

The method of 5, wherein the method includes receiving the selection input from at least one of a microphone, a button, food pedal, joystick, stylus, touch input, or eye tracker.

### Claim 7 (depends on 1)

The method of claim 1, further comprising
positioning a second surgical instrument in a body cavity;
mounting the second surgical instrument to a second robotic manipulator arm; and
causing at least one of the following:
causing the first robotic manipulator to move a tip of the first surgical instrument to a location a predetermined standoff distance from the tagged region of interest, or
causing the second robotic manipulator to move a tip of the second surgical instrument to a location a predetermined standoff distance from the tagged region of interest.

### Claim 8 (depends on 1)

The method of claim 1, further comprising
mounting the endoscope to the second robotic manipulator arm; and
causing the second robotic manipulator to move the endoscope towards the tagged region of interest.

### Claim 9 (depends on 1)

The method of claim 1, wherein the region of interest identified by the tag has a depth, and wherein the method further comprising:
positioning a second surgical instrument in a body cavity;
mounting the second surgical instrument to a second robotic manipulator arm; and
preventing movement of at least one of the first or second instruments to a location in the body cavity that is deeper than the depth.

### Claim 10 (depends on 1)

The method of claim 1, further comprising detecting in real-time a change in the image during the medical procedure, the change indicative of the feature of interest becoming at least partialy obscured by smoke or pooling blood; and
altering a quality of the displayed graphical tag to enhance its visibility in response to the detected change.
