# US 12,002,571 B2 · Dynamic surgical visualization systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,571 B2
- Document ID: US12002571B2
- Status: Granted
- Kind: B2
- Application number: 16/729,747
- Filed: 2019-12-30
- Published: 2021-07-01
- Granted: 2024-06-04
- Earliest claimed: 2019-12-30
- Assignees: Cilag GmbH International
- Applicants: Cilag GmbH International
- Inventors: Frederick E. Shelton, IV; Jason L. Harris
- Web page: https://patentgrep.com/patent/US12002571B2

## CPC

- G16H 30/40: for processing medical images, e.g. editing
- A61B 1/000094: {extracting biological structures}
- A61B 1/0638: {providing two or more wavelengths}
- A61B 1/0646: {with illumination filters}
- A61B 18/1445: {at the distal end of a shaft, e.g. forceps or scissors at the end of a rigid rod}
- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- A61B 34/25: {User interfaces for surgical systems}
- A61B 90/361: {Image-producing devices, e.g. surgical cameras}
- G01B 11/026: {by measuring distance between sensor and object (G01B11/0608 takes precedence)}
- G01B 11/2513: {with several lines being projected in more than one direction, e.g. grids, patterns}
- G06T 15/20: Perspective computation
- G16H 20/40: relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
- A61B 5/02: Detecting, measuring or recording for evaluating the cardiovascular system, e.g. pulse, heart rate, blood pressure or blood flow
- A61B 17/320068: {using mechanical vibrations, e.g. ultrasonic (A61B17/22012 takes precedence; dental tooth drilling devices operated by vibration A61C3/03; removing intra-ocular material using mechanical vibrations A61F9/00745)}
- A61B 18/14: Probes or electrodes therefor
- A61B 34/30: Surgical robots
- A61B 2017/00809: {Lung operations}
- A61B 2018/00601: {Cutting}
- A61B 2034/2051: {Electromagnetic tracking systems}
- A61B 2034/2065: {Tracking using image or pattern recognition}
- A61B 2034/302: {specifically adapted for manipulations within body cavities, e.g. within abdominal or thoracic cavities}
- A61B 2090/304: {using chemi-luminescent materials}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/367: {creating a 3D dataset from 2D images using position information}
- A61B 2090/371: {with simultaneous use of two cameras}
- A61B 2090/372: {Details of monitor hardware}
- A61B 2090/373: {using light, e.g. by using optical scanners}
- A61B 2090/374: {NMR or MRI}
- A61B 2090/3762: {using computed tomography systems [CT]}
- A61B 2090/3941: {Photoluminescent markers}
- G01J 2003/2826: {Multispectral imaging, e.g. filter imaging}
- G01S 17/894: Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2210/41: Medical
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G01J: MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY (light sources F21, H01J, H01K, H05B; investigating properties of materials by optical means G01N)
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES

## Abstract

Various systems and methods of controlling a surgical visualization system and controlling devices according to the displayed point of view of the surgical visualization system are disclosed. The visualization system can allow users to change, update, or manipulate the displayed visualization utilizing overlapping or multiple images. Further, the controls and/or display screens of surgical instruments coupled to the visualization system can be adjusted to correspond to the orientation of the surgical instrument as visualized by the visualization system.

## Claims

### Claim 1

A control system for a handheld surgical instrument comprising a user control, the control system comprising:
an imaging system, comprising:
an emitter configured to emit electromagnetic radiation (EMR); and
an image sensor configured to receive the EMR reflected from a surgical site; and
a control circuit in signal communication with the imaging system and the handheld surgical instrument, wherein the control circuit is configured to:
generate an image of the surgical site based on the EMR received by the image sensor;
define a first coordinate system with respect to the surgical site according to the image;
receive a second coordinate system defined by the handheld surgical instrument;
determine a transfer function to translate a coordinate in the second coordinate system to the first coordinate system; and
provide the transfer function to the handheld surgical instrument to cause the handheld surgical instrument to adjust the user control according to the transfer function.

### Claim 2 (depends on 1)

The control system of claim 1, further comprising a surgical hub, the surgical hub comprising the control circuit.

### Claim 3 (depends on 2)

The control system of claim 2, wherein the surgical hub comprises a situational awareness module configured to provide the transfer function to the handheld surgical instrument according to a surgical context determined by the situational awareness module.

### Claim 4 (depends on 1)

The control system of claim 1, wherein the transfer function is configured to cause the handheld surgical instrument to adjust at least one articulation control actuator of the handheld surgical instrument.

### Claim 5 (depends on 4)

The control system of claim 4, wherein the handheld surgical instrument comprises an end effector, a shaft, and an articulation joint, wherein the end effector is articulatable relative to the shaft about the articulation joint, wherein the at least one articulation control actuator is configured to articulate the end effector about the articulation joint in a first direction, and wherein the at least one articulation control actuator is configured to articulate the end effector about the articulation joint in a second direction different than the first direction based on the handheld surgical instrument being provided with the transfer function.

### Claim 6 (depends on 1)

The control system of claim 1, wherein the transfer function is configured to cause the handheld surgical instrument to adjust a graphical user interface of the handheld surgical instrument.

### Claim 7 (depends on 6)

The control system of claim 6, wherein the graphical user interface is configured to display an image on the graphical user interface in a first orientation, and wherein the graphical user interface is configured to display the image on the graphical user interface in a second orientation different than the first orientation based on the handheld surgical instrument being provided with the transfer function.

### Claim 8

A control system, comprising:
an imaging system configured to capture a live feed of a surgical site from a first point of view, wherein the imaging system comprises:
an emitter configured to emit structured electromagnetic radiation (EMR); and
an image sensor configured to receive the structured EMR reflected from the surgical site;
a display screen; and
a control circuit communicatively coupled to the imaging system and the display screen, wherein the control circuit is configured to:
generate a first image of the surgical site based on the structured EMR received by the image sensor;
receive a second image of the surgical site;
align the first image and the second image;
generate a three-dimensional representation of the surgical site based on the first image and the second image as aligned;
display the live feed of the surgical site from the first point of view on the display screen, wherein the live feed is displayed with a first visual characteristic;
receive a user selection to manipulate a point of view of the surgical site; and
according to the user selection, update the point of view as displayed on the display screen from a first state, in which the live feed of the surgical site from the first point of view is displayed on the display screen with the first visual characteristic, to a second state, in which a second point of view of the surgical site is displayed on the display screen, wherein, in the second state, an out-of-sight portion of the surgical site from the first point of view is shown on the display screen, wherein the out-of-sight portion is displayed based on the three-dimensional representation of the surgical site, and wherein the out-of-sight portion is displayed with a second visual characteristic different than the first visual characteristic.

### Claim 9 (depends on 8)

The control system of claim 8, wherein the control circuit is further configured to isolate an irregularity from at least one of the first image or the second image based on the other of the first image or the second image, and wherein the three-dimensional representation generated from the first image and the second image lacks the irregularity.

### Claim 10 (depends on 9)

The control system of claim 9, wherein the irregularity comprises at least one of a tissue, a biological structure, or an object.

### Claim 11 (depends on 8)

The control system of claim 8, wherein the control circuit is further configured to:
generate a third image of the surgical site based on the structured EMR received by the image sensor; and
update the three-dimensional representation based on the third image.

### Claim 12 (depends on 8)

The control system of claim 8, wherein the second image is received from a non-real-time image source.

### Claim 13 (depends on 12)

The control system of claim 12, wherein the non-real-time image source comprises at least one of a CT scan or an MRI.

### Claim 14 (depends on 8)

The control system of claim 8, wherein the display screen comprises an input interface, and wherein the control circuit is configured to receive the user selection from the input interface.

### Claim 15 (depends on 14)

The control system of claim 14, wherein the input interface comprises a manipulatable widget.

### Claim 16 (depends on 14)

The control system of claim 14, wherein the display further comprises a coordinate element configured to visually communicate a shift from a coordinate system of the first state to a coordinate system of second state.

### Claim 17 (depends on 8)

The control system of claim 8, wherein the user selection comprises an input to rotate the three-dimensional representation from the first point of view to the second point of view.

### Claim 18 (depends on 8)

The control system of claim 8, wherein the first visual characteristic comprises a first color, and wherein the second visual characteristic comprises a second color different than the first color.

### Claim 19 (depends on 8)

The control system of claim 8, wherein the second visual characteristic comprises highlighting.