# US 11,992,200 B2 · Instrument control surgical imaging systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,992,200 B2
- Document ID: US11992200B2
- Status: Granted
- Kind: B2
- Application number: 17/449,769
- Filed: 2021-10-01
- Published: 2023-03-30
- Granted: 2024-05-28
- Earliest claimed: 2021-09-29
- Assignees: Cilag GmbH International
- Applicants: Cilag GmbH International
- Inventors: Frederick E. Shelton, IV; Charles J. Scheib; Jason L. Harris
- Web page: https://patentgrep.com/patent/US11992200B2

## CPC

- A61B 17/0218: {for minimally invasive surgery (A61B17/0281, A61B17/0469, A61B17/12013, A61B17/1285, A61B17/29, A61B17/320016 take precedence)}
- A61B 17/3423: {Access ports, e.g. toroid shape introducers for instruments or hands (access sites for liquids A61M39/0247)}
- A61B 1/00004: {characterised by electronic signal processing}
- A61B 1/00006: {of control signals}
- A61B 1/00009: {of image signals during a use of endoscope}
- A61B 1/000094: {extracting biological structures}
- A61B 1/000095: {for image enhancement}
- A61B 1/00045: {Display arrangement}
- A61B 1/00048: {Constructional features of the display}
- A61B 1/0005: {combining images e.g. side-by-side, superimposed or tiled}
- A61B 1/00055: {for alerting the user}
- A61B 1/00087: {Tools (A61B17/00234 takes precedence)}
- A61B 1/00097: {Sensors}
- A61B 1/00126: {optical, e.g. for light supply cables}
- A61B 1/00149: {using articulated arms}
- A61B 1/00154: {using guiding arrangements for insertion}
- A61B 1/0016: {using motor drive units}
- A61B 1/00193: {adapted for stereoscopic vision}
- A61B 1/00194: {adapted for three-dimensional imaging}
- A61B 1/009: {with bending or curvature detection of the insertion part}
- A61B 1/018: for receiving instruments
- A61B 1/043: {for fluorescence imaging}
- A61B 1/046: {for infrared imaging}
- A61B 1/053: {being detachable}
- A61B 1/0605: {for spatially modulated illumination}
- A61B 1/0638: {providing two or more wavelengths}
- A61B 1/0676: {at distal tip of an endoscope}
- A61B 1/07: using light-conductive means, e.g. optical fibres
- A61B 1/3132: {for laparoscopy}
- A61B 5/0084: {for introduction into the body, e.g. by catheters (A61B5/1459 takes precedence)}
- A61B 5/489: {Blood vessels}
- A61B 5/4893: {Nerves}
- A61B 17/00234: {for minimally invasive surgery (A61B17/0218, A61B17/0469, A61B17/12013, A61B17/1285, A61B17/29, A61B17/320016 take precedence)}
- A61B 17/12099: {characterised by the location of the occluder}
- A61B 17/12104: {in an air passage}
- A61B 17/12136: {Balloons}
- A61B 17/3421: {Cannulas}
- A61B 17/3474: {Insufflating needles, e.g. Veress needles}
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 34/30: Surgical robots
- A61B 90/06: {Measuring instruments not otherwise provided for}
- A61B 90/361: {Image-producing devices, e.g. surgical cameras}
- A61B 90/37: {Surgical systems with images on a monitor during operation}
- A61B 90/39: {Markers, e.g. radio-opaque or breast lesions markers}
- A61B 90/50: Supports for surgical instruments, e.g. articulated arms
- G01B 11/026: {by measuring distance between sensor and object (G01B11/0608 takes precedence)}
- G01B 11/2513: {with several lines being projected in more than one direction, e.g. grids, patterns}
- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- G06T 7/74: {involving reference images or patches}
- H04N 13/111: Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
- H04N 13/156: Mixing image signals
- A61B 5/0064: {Body surface scanning}
- A61B 5/0075: {by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy (A61B5/0071 takes precedence)}
- A61B 5/7425: {Displaying combinations of multiple images regardless of image source, e.g. displaying a reference anatomical image with a live image}
- A61B 17/1155: {Circular staplers comprising a plurality of staples}
- A61B 17/320068: {using mechanical vibrations, e.g. ultrasonic (A61B17/22012 takes precedence; dental tooth drilling devices operated by vibration A61C3/03; removing intra-ocular material using mechanical vibrations A61F9/00745)}
- A61B 17/3462: {with means for changing the diameter or the orientation of the entrance port of the cannula, e.g. for use with different-sized instruments, reduction ports, adapter seals (expandable cannulas A61B17/3417)}
- A61B 18/14: Probes or electrodes therefor
- A61B 34/32: operating autonomously
- A61B 34/37: Leader-follower robots (A61B34/35 takes precedence)
- A61B 90/57: Accessory clamps
- A61B 2017/00022: {Sensing or detecting at the treatment site}
- A61B 2017/00057: {Light}
- A61B 2017/00061: {spectrum}
- A61B 2017/00075: {Motion}
- A61B 2017/00199: {with a console, e.g. a control panel with a display}
- A61B 2017/00292: {mounted on or guided by flexible, e.g. catheter-like, means}
- A61B 2017/00398: {using powered actuators, e.g. stepper motors, solenoids}
- A61B 2017/00809: {Lung operations}
- A61B 2017/00818: {Treatment of the gastro-intestinal system}
- A61B 2017/00876: {magnetic}
- A61B 2017/3419: {Sealing means between cannula and body}
- A61B 2017/3445: {used as instrument channel for multiple instruments}
- A61B 2017/3466: {for simultaneous sealing of multiple instruments}
- A61B 2017/347: {Locking means, e.g. for locking instrument in cannula}
- A61B 2017/3484: {Anchoring means, e.g. spreading-out umbrella-like structure}
- A61B 2018/00702: {Power or energy}
- A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones}
- A61B 2034/2051: {Electromagnetic tracking systems}
- A61B 2034/2055: {Optical tracking systems}
- A61B 2034/2057: {Details of tracking cameras}
- A61B 2034/2061: {using shape-sensors, e.g. fiber shape sensors with Bragg gratings}
- A61B 2034/2065: {Tracking using image or pattern recognition}
- A61B 2034/301: {for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes}
- A61B 2090/061: {for measuring dimensions, e.g. length}
- A61B 2090/064: {for measuring force, pressure or mechanical tension}
- A61B 2090/364: {Correlation of different images or relation of image positions in respect to the body}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/371: {with simultaneous use of two cameras}
- A61B 2090/3937: {Visible markers}
- A61B 2090/3945: {Active visible markers, e.g. light emitting diodes}
- A61B 2090/3954: {magnetic, e.g. NMR or MRI}
- A61B 2505/05: Surgical care
- A61B 2576/00: Medical imaging apparatus involving image processing or analysis (A61B1/00009, A61B6/52 and A61B8/52 take precedence)
- G01B 11/25: by projecting a pattern, e.g. {one or more lines,} moiré fringes on the object (G01B11/255 takes precedence {; image analysis for depth or shape recovery G06T7/50})
- G06T 2207/10068: Endoscopic image
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Surgical systems are provided. The surgical system includes a first scope device configured to transmit image data of a first scene within a field of view of the first scope device. A second scope device is configured to transmit image data of a second scene within a field of view of the second scope device. A tracking device is associated with one of the scope devices and configured to transmit a signal indicative of a location of the one of the scope devices. A controller is configured to receive the transmitted image data and the transmitted signal to determine a relative distance between the scope devices and to provide a merged image of at least a portion of the scope devices in a single scene, where at least one of the scope devices in the merged image is a representative depiction.

## Claims

### Claim 1

A surgical system, comprising:
a first scope device configured to be at least partially disposed within at least one of a natural body lumen and an organ and configured to transmit image data of a first scene within a field of view of the first scope device;
a second scope device configured to be at least partially disposed outside of the at least one of the natural body lumen and the organ and to transmit image data of a second scene within a field of view of the second scope device, the second scene being different than the first scene;
a tracking device associated with one of the first scope device or the second scope device and configured to transmit a signal indicative of a location of the one of the first scope device or the second scope device relative to the other one of the first scope device or the second scope device; and
a controller configured to receive (i) the transmitted image data of the first and second scenes and (ii) the transmitted signal, to determine, based on the transmitted signal, a relative distance between the first scope device and the second scope device, and to provide, based on the transmitted image data and relative distance between the first and second scopes, a merged image of at least a portion of at least the first scope device and the second scope device in a single scene, wherein at least one of the first scope device and the second scope device in the merged image is a representative depiction thereof.

### Claim 2 (depends on 1)

The surgical system of claim 1, wherein the first scope device and the second scope device are each illustrated as a representative depiction thereof in the merged image.

### Claim 3 (depends on 1)

The surgical system of claim 1, further comprising a first display that is configured to display the first scene and a second display that is configured to display the second scene.

### Claim 4 (depends on 3)

The surgical system of claim 3, wherein at least one of the first display and the second display is further configured to display the single scene.

### Claim 5 (depends on 3)

The surgical system of claim 3, further comprising a third display that is configured to display the single scene.

### Claim 6 (depends on 1)

The surgical system of claim 1, wherein the first scope device is an endoscope and the second scope device is a laparoscope.

### Claim 7 (depends on 1)

The surgical system of claim 1, wherein the first scene does not include the second scope device, and wherein the second scene does not include the first scope device.

### Claim 8 (depends on 1)

The surgical system of claim 1, wherein the tracking device is associated with the first scope device.

### Claim 9 (depends on 1)

The surgical system of claim 1, wherein the tracking device is associated with the second scope device.

### Claim 10 (depends on 1)

The surgical system of claim 1, wherein the signal is further indicative of an orientation of the first scope device within one of the natural body lumen and the organ relative to the second scope device, and wherein the controller is further configured to determine, based on the transmitted signal, a relative orientation of the first scope device.

### Claim 11 (depends on 1)

The surgical system of claim 1, wherein the signal is further indicative of an orientation of the second scope device positioned outside of the at least one of the natural body lumen and the organ relative to the first scope device, and wherein the controller is further configured to determine, based on the transmitted signal, a relative orientation of the second scope device.

### Claim 12 (depends on 1)

The surgical system of claim 1, the controller is further configured to determine, based on at least the transmitted image data, at least one of a location and an orientation of at least one instrument positioned outside of the at least one natural body lumen and the organ relative to the first scope device, wherein at least a portion of the at least one instrument is illustrated as an actual depiction or representative depiction thereof in the merged image.

### Claim 13 (depends on 1)

Surgical system of claim 1, wherein the controller is further configured to (i) receive an additional signal that is indicative of at least one of a location and an orientation of at least one instrument positioned outside of the at least one natural body lumen and the organ relative to the second scope device, (ii) to determine, based on the transmitted additional signal, at least one of a relative location and a relative orientation of the at least one instrument, wherein at least a portion of the at least one instrument is illustrated as an actual depiction or representative depiction thereof in the merged image.

### Claim 14

A method, comprising:
transmitting, by a first scope device, image data of a first scene within a field of view of the first scope device while at least a portion of the first device is positioned within at least one of a natural body lumen and an organ;
transmitting, by a second scope device, image data of a second scene within a field of view of the second scope device while the second scope device is positioned outside of the at least one of the natural body lumen and the organ, the second scene being different than the first scene;
transmitting, by a tracking device, a signal indicative of a location of one of the positioned first scope device or second scope device relative to the other positioned first scope device or second scope device;
receiving, by a controller, the transmitted image data of the first and second scenes and the transmitted signals of the location of the first and second scope devices;
determining, by the controller and based on the transmitted signal, a relative distance between the first scope device and the second scope device; and
generating, by the controller and based on the transmitted image data and the relative distance between the first and second scope devices, a merged image of at least a portion of at least the first scope device and the second scope device in a single scene, wherein at least one of the first scope device and the second scope device in the single scene is a representative depiction thereof.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
illustrating a representative depiction of the first scope device in the merged image; and
illustrating a representative depiction of the second scope device in the merged image.

### Claim 16 (depends on 14)

The method of claim 14, further comprising:
displaying the first scene on a first display; and
displaying the second scene on a second display.

### Claim 17 (depends on 14)

The method of claim 14, further comprising displaying the single scene on at least one of the first display and the second display.

### Claim 18 (depends on 14)

The method of claim 14, further comprising displaying the single scene on a third display.

### Claim 19 (depends on 14)

The method of claim 14, further comprising transmitting, by the tracking device, a signal indicative of an orientation of the one of the first scope device or the second scope device relative to the other one of the first scope device or the second scope device.

### Claim 20 (depends on 14)

The method of claim 14, further comprising:
determining, by the controller and based on the transmitted image data, at least one of a position and an orientation of at least one or more instruments positioned outside of the at least one natural body lumen and the organ relative to the first scope device.