# US 2025/0166787 A1 · Visualization of an internal process of an automated operation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0166787 A1
- Document ID: US20250166787A1
- Status: Published application
- Kind: A1
- Application number: 18/954,195
- Filed: 2024-11-20
- Published: 2025-05-22
- Earliest claimed: 2023-11-22
- Applicants: Cilag GmbH International
- Inventors: Frederick E. Shelton, IV; John E. Brady; Shannon L. Jones; Laura S. Downing; Andrew C. Deck; Weston Staney Hirschfeld; Scotty A. Chung; Anthony J. Evers; Todd M. Dahling; Aaron Crooker; Christopher Johns; Devin J. Clark; Douglas W. Fisher; Madeleine Jayme
- Web page: https://patentgrep.com/patent/US20250166787A1

## CPC

- G16H 20/40: relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- A61B 34/25: {User interfaces for surgical systems}
- A61B 34/30: Surgical robots
- A61B 34/32: operating autonomously
- A61B 34/37: Leader-follower robots (A61B34/35 takes precedence)
- A61B 34/76: {Manipulators having means for providing feel, e.g. force or tactile feedback}
- B25J 9/1689: {Teleoperation}
- G05B 15/02: electric
- G06F 9/542: {Event management; Broadcasting; Multicasting; Notifications}
- G16H 10/60: for patient-specific data, e.g. for electronic patient records
- G16H 30/40: for processing medical images, e.g. editing
- G16H 40/40: for the management of medical equipment or devices, e.g. scheduling maintenance or upgrades
- G16H 40/60: for the operation of medical equipment or devices
- G16H 40/63: for local operation
- G16H 40/67: for remote operation
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G16H 50/30: for calculating health indices; for individual health risk assessment
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G16H 70/20: relating to practices or guidelines
- H04L 67/12: specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
- A61B 2034/101: {Computer-aided simulation of surgical operations}
- A61B 2034/107: {Visualisation of planned trajectories or target regions}
- A61B 2034/2048: {using an accelerometer or inertia sensor}
- A61B 2034/252: {indicating steps of a surgical procedure}
- A61B 2034/254: {being adapted depending on the stage of the surgical procedure}
- A61B 2034/301: {for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes}
- A61B 2034/302: {specifically adapted for manipulations within body cavities, e.g. within abdominal or thoracic cavities}
- A61B 2034/305: {Details of wrist mechanisms at distal ends of robotic arms}
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Device and methods for visualizing internal processes of an automated operation. An example device may receive an external data stream from a source external to the surgical system. The device may derive, based at least on the external data stream, decision contextual information. The device may select a surgical option associated with a surgical instrument based on the decision context information. The device may generate a visual indication of the decision context information associated with selecting the surgical option. The device may generate a control signal associated with the surgical instrument based on the selected surgical option.

## Claims

### Claim 1

A surgical system comprising:
a processor configured to:
receive an external data stream from a source external to the surgical system;
derive, based at least on the external data stream, decision contextual information;
select a surgical option associated with a surgical instrument based on the decision context information;
generate a visual indication of the decision context information associated with selecting the surgical option; and
generate a control signal associated with the surgical instrument based on the selected surgical option.

### Claim 2 (depends on 1)

The surgical system of claim 1, wherein the surgical option is associated with stone removal, and the external data stream comprises a visualization of a patient's organ, and the processor is further configured to:
identify, based on the visualization of the patient's organ, a potential perimeter of a stone in the patient's organ, wherein the decision context information comprises the identified potential perimeter of the stone in the patient's organ, and the visual indication of the decision context information comprises a visual indication of the identified potential perimeter of the stone in the patient's organ.

### Claim 3 (depends on 1)

The surgical system of claim 1, wherein the surgical option is associated with stone removal, and the external data stream comprises a visualization of a patient's organ, and the processor is further configured to:
identify, based on the visualization of the patient's organ, a potential perimeter of a stone in the patient's organ; and
select a stone removal treatment location based on the identified potential perimeter of the stone in the patient's organ, wherein the control signal associated with the surgical instrument is generated based on the selected stone removal treatment location, and the visual indication of the decision context information comprises a visual indication of the identified potential perimeter of the stone in the patient's organ.

### Claim 4 (depends on 1)

The surgical system of claim 1, wherein the processor is further configured to:
obtain a plurality of surgical options associated with the surgical instrument; and
determine, based at least on the external data stream, respective system confidence assessments that correspond to the plurality of surgical options, wherein the decision context information comprises the respective system confidence assessments that correspond to the plurality of surgical options, and the visual indication of the decision context information comprises the system confidence assessment that corresponds to the selected surgical option.

### Claim 5 (depends on 1)

The surgical system of claim 1, wherein the external data stream comprises a visualization of a patient's organ, and the processor is further configured to:
determine, based on the external data stream and the selected surgical option, a resultant visualization of the patient's organ, wherein the visual indication of the decision context information comprises a visualization of the patient's organ pre-therapy and the resultant visualization of the patient's organ post-therapy.

### Claim 6 (depends on 1)

The surgical system of claim 1, wherein the surgical option is associated with stone removal, and the external data stream comprises a visualization of a patient's organ, and the processor is further configured to:
identify, based on the visualization of the patient's organ, a first potential perimeter of a stone in the patient's organ and a second potential perimeter of the stone in the patient's organ, the second potential perimeter encompassing the first potential perimeter; and
calculate a first system confidence percentage associated with the first potential perimeter and a second system confidence percentage associated with the second potential perimeter, wherein the decision context information comprises the first system confidence percentage and the second system confidence percentage, and wherein the visual indication of the decision context information comprises a visual indication of the first potential perimeter of the stone and its associated first system confidence percentage, and a visual indication of the first potential perimeter of the stone and its associated second system confidence percentage.

### Claim 7 (depends on 1)

The surgical system of claim 1, wherein the surgical option is associated with stone removal, and the external data stream comprises a visualization of a patient's organ, and the processor is further configured to:
identify, based on the visualization of the patient's organ, a potential perimeter of a stone in the patient's organ;
identify a plurality of potential stone removal treatment locations;
determine, based on the potential perimeter of the stone in the patient's organ, respective confidence assessments that correspond to the plurality of potential stone removal treatment locations; and
select a stone removal treatment location from the potential stone removal treatment locations based on their respective confidence assessments, wherein the control signal associated with the surgical instrument is generated based on the selected stone removal treatment location, and the visual indication of the decision context information comprises a visual indication of the potential perimeter of the stone in the patient's organ.

### Claim 8

A method, performed by a surgical system, the method comprising:
receiving an external data stream from a source external to the surgical system;
deriving, based at least on the external data stream, decision contextual information;
selecting a surgical option associated with a surgical instrument based on the decision context information;
generating a visual indication of the decision context information associated with selecting the surgical option; and
generating a control signal associated with the surgical instrument based on the selected surgical option.

### Claim 9 (depends on 8)

The method of claim 8, wherein the surgical option is associated with stone removal, and the external data stream comprises a visualization of a patient's organ, and the method further comprises:
identifying, based on the visualization of the patient's organ, a potential perimeter of a stone in the patient's organ, wherein the decision context information comprises the identified potential perimeter of the stone in the patient's organ, and the visual indication of the decision context information comprises a visual indication of the identified potential perimeter of the stone in the patient's organ.

### Claim 10 (depends on 8)

The method of claim 8, wherein the surgical option is associated with stone removal, and the external data stream comprises a visualization of a patient's organ, and the method further comprises:
identifying, based on the visualization of the patient's organ, a potential perimeter of a stone in the patient's organ; and
selecting a stone removal treatment location based on the identified potential perimeter of the stone in the patient's organ, wherein the control signal associated with the surgical instrument is generated based on the selected stone removal treatment location, and the visual indication of the decision context information comprises a visual indication of the identified potential perimeter of the stone in the patient's organ.

### Claim 11 (depends on 8)

The method of claim 8, wherein the method further comprises:
obtaining a plurality of surgical options associated with the surgical instrument; and
determining, based at least on the external data stream, respective system confidence assessments that correspond to the plurality of surgical options, wherein the decision context information comprises the respective system confidence assessments that correspond to the plurality of surgical options, and the visual indication of the decision context information comprises the system confidence assessment that corresponds to the selected surgical option.

### Claim 12 (depends on 8)

The method of claim 8, wherein the external data stream comprises a visualization of a patient's organ, and the method further comprises:
determining, based on the external data stream and the selected surgical option, a resultant visualization of the patient's organ, wherein the visual indication of the decision context information comprises a visualization of the patient's organ pre-therapy and the resultant visualization of the patient's organ post-therapy.

### Claim 13 (depends on 8)

The method of claim 8, wherein the surgical option is associated with stone removal, and the external data stream comprises a visualization of a patient's organ, and the method further comprises:
identifying, based on the visualization of the patient's organ, a first potential perimeter of a stone in the patient's organ and a second potential perimeter of the stone in the patient's organ, the second potential perimeter encompassing the first potential perimeter; and
calculating a first system confidence percentage associated with the first potential perimeter and a second system confidence percentage associated with the second potential perimeter, wherein the decision context information comprises the first system confidence percentage and the second system confidence percentage, and wherein the visual indication of the decision context information comprises a visual indication of the first potential perimeter of the stone and its associated first system confidence percentage, and a visual indication of the first potential perimeter of the stone and its associated second system confidence percentage.

### Claim 14 (depends on 8)

The method of claim 8, wherein the surgical option is associated with stone removal, and the external data stream comprises a visualization of a patient's organ, and the method further comprises:
identifying, based on the visualization of the patient's organ, a potential perimeter of a stone in the patient's organ;
identifying a plurality of potential stone removal treatment locations;
determining, based on the potential perimeter of the stone in the patient's organ, respective confidence assessments that correspond to the plurality of potential stone removal treatment locations; and
selecting a stone removal treatment location from the potential stone removal treatment locations based on their respective confidence assessments, wherein the control signal associated with the surgical instrument is generated based on the selected stone removal treatment location, and the visual indication of the decision context information comprises a visual indication of the potential perimeter of the stone in the patient's organ.

### Claim 15

A surgical system comprising:
a processor configured to:
receive an external data stream associated with a patient's organ;
obtain a plurality of surgical options associated with a surgical instrument;
determine, based at least on the external data stream, respective system confidence assessments that correspond to the plurality of surgical options;
select, from a plurality of surgical options, a surgical option based on their respective system confidence assessments;
generate a control signal associated with the surgical instrument based on the selected surgical option; and
generate a visual indication of the system confidence assessment that corresponds to the selected surgical option.

### Claim 16 (depends on 15)

The surgical system of claim 15, wherein the surgical options are associated with stone removal, and the external data stream comprises a visualization of the patient's organ, and the processor is further configured to:
identify, based on the visualization of the patient's organ, a potential perimeter of a stone in the patient's organ;
identify a plurality of potential stone removal treatment locations as the plurality of surgical options;
determine, based on the potential perimeter of the stone in the patient's organ, respective confidence assessments that correspond to the plurality of potential stone removal treatment locations; and
select a stone removal treatment location from the plurality of potential stone removal treatment locations based on their respective confidence assessments, wherein the control signal associated with the surgical instrument is generated based on the selected stone removal treatment location, and the visual indication of the system confidence assessment comprises a visual indication of the potential perimeter of the stone in the patient's organ.

### Claim 17 (depends on 15)

The surgical system of claim 15, wherein the surgical options are associated with stone removal, and the processor is further configured to:
identify, based on the external data stream, a first potential perimeter of a stone in the patient's organ and a second potential perimeter of the stone in the patient's organ, the second potential perimeter encompassing the first potential perimeter; and
calculate a first system confidence percentage associated with the first potential perimeter and a second system confidence percentage associated with the second potential perimeter, wherein the visual indication of the system confidence assessment comprises a visual indication of the first potential perimeter of the stone and its associated first system confidence percentage and a visual indication of the first potential perimeter of the stone and its associated second system confidence percentage.
