# US 9,775,499 B2 · Augmented visualization for a surgical robot using a captured visible image combined with a fluorescence image and a captured visible image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,775,499 B2
- Document ID: US9775499B2
- Status: Granted
- Kind: B2
- Application number: 14/332,684
- Filed: 2014-07-16
- Published: 2015-03-19
- Granted: 2017-10-03
- Earliest claimed: 2008-04-26
- Assignees: Intuitive Surgical Operations, Inc.
- Applicants: Intuitive Surgical Operations, Inc.
- Inventors: David Scott; Wenyi Zhao; Brian D. Hoffman; John D. Stern; Tao Zhao
- Web page: https://patentgrep.com/patent/US9775499B2

## CPC

- A61B 1/043: {for fluorescence imaging}
- A61B 1/000096: {using artificial intelligence}
- A61B 1/0005: {combining images e.g. side-by-side, superimposed or tiled}
- A61B 1/00149: {using articulated arms}
- A61B 1/00186: {with imaging filters}
- A61B 1/00193: {adapted for stereoscopic vision}
- A61B 1/00194: {adapted for three-dimensional imaging}
- A61B 1/05: characterised by the image sensor, e.g. camera, being in the distal end portion
- A61B 1/0638: {providing two or more wavelengths}
- A61B 1/313: for introducing through surgical openings, e.g. laparoscopes
- A61B 5/0071: {by measuring fluorescence emission}
- A61B 5/0084: {for introduction into the body, e.g. by catheters (A61B5/1459 takes precedence)}
- A61B 34/30: Surgical robots
- A61B 90/361: {Image-producing devices, e.g. surgical cameras}
- H04N 13/207: using a single two-dimensional [2D] image sensor
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- A61B 1/063: {for monochromatic or narrow-band illumination}
- A61B 5/0075: {by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy (A61B5/0071 takes precedence)}
- A61B 34/37: Leader-follower robots (A61B34/35 takes precedence)
- A61B 90/30: Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure
- A61B 2034/301: {for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes}
- A61B 2090/364: {Correlation of different images or relation of image positions in respect to the body}
- G01N 21/6456: {Spatial resolved fluorescence measurements; Imaging}
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q)

## Abstract

An endoscope with an optical channel is held and positioned by a robotic surgical system. A capture unit captures (1) a visible first image at a first time and (2) a visible second image combined with a fluorescence image from the light at a second time. An image processing system receives (1) the visible first image and (2) the visible second image combined with the fluorescence image and generates at least one fluorescence image. A display system outputs an output image including an artificial fluorescence image.

## Claims

### Claim 1

A surgical system comprising:
an endoscope held and positioned by the surgical system, the endoscope comprising:
an optical channel for transporting light from tissue;
a capture unit coupled to the optical channel, the capture unit being configured to capture:
(1) at a first time, a first frame from the light, the first frame including a first visible image, the first visible image comprising a first plurality of visible color components; and
(2) at a second time different from the first time, a second frame from the light, the second frame including a combination image, the combination image being a combination of a second visible image and a first fluorescence image, the combination image comprising a second plurality of color components, wherein one color component of the second plurality of color component includes one color component that is captured as a combination of the first fluorescence image and a visible color component, image and other color components of the second plurality of color components are captured as visible color components different from the visible color component of the one color component;
an image processing system coupled to said capture unit, the image processing system comprising:
a fluorescence image with artifacts generator coupled to receive the first and second frames, the fluorescence image with artifacts generator being configured to create a second fluorescence image including artifacts from the first frame and the second frame; and
a fluorescence image extractor coupled to the fluorescence image with artifacts generator, the fluorescence image extractor being configured to create a third fluorescence image based on the second fluorescence image including artifacts; and
a display system coupled to the image processing system, the display system being configured to output an image including an artificial fluorescence image, the artificial fluorescence image being based on the third fluorescence image.

### Claim 2 (depends on 1)

The surgical system of claim 1, the output image further comprising:
a blend of the first visible image and the artificial fluorescence image.

### Claim 3 (depends on 1)

The surgical system of claim 1, the display system further comprising:
a blend module coupled to receive the first frame and to receive the artificial fluorescence image; and
a display unit coupled to the blend module.

### Claim 4 (depends on 3)

The surgical system of claim 3, wherein the display system further comprises:
a display mode select coupled to the blend module, the display mode select having a plurality of selectable modes,
wherein upon selection of a first mode of the plurality of selectable modes; the display unit outputs only the first visible image; and
wherein upon selection of a second mode of the plurality of selectable modes, the display unit outputs the image including the artificial fluorescence image.

### Claim 5 (depends on 1)

The surgical system of claim 1, wherein the endoscope further comprises an illumination channel, and wherein the surgical system further comprises:
a combination light source coupled to the illumination channel; the combination light source comprising:
a first light source configured to provide a first illumination light; and
a second light source configured to provide a second illumination light,
wherein the second illumination light is different from the first illumination light; and
wherein said second illumination light is synchronized with the capture of the second frame.

### Claim 6 (depends on 1)

The surgical system of claim 1, wherein said image processing system further comprises a fluorescence image enhancement system coupled to said fluorescence image extractor, the fluorescence image enhancement system being configured to receive said third fluorescence image and being configured to create said artificial fluorescence image from the third fluorescence image.

### Claim 7 (depends on 1)

The surgical system of claim 1, the fluorescence image with artifacts generator being configured to subtract the first frame from the second frame to generate a plurality of frame-to-frame color component differences, wherein one of the frame-to-frame color component differences is the second fluorescence image including artifacts.

### Claim 8 (depends on 7)

The surgical system of claim 7, the image processing system further comprising:
a scale module coupled between the fluorescence image with artifacts generator and the fluorescence image extractor, the scale module being configured to normalize the plurality of frame-to-frame color component differences.

### Claim 9 (depends on 8)

The surgical system of claim 8, the fluorescence image extractor being configured to perform cross-channel subtraction using the normalized plurality of frame-to-frame color component differences to obtain the third fluorescence image.

### Claim 10 (depends on 1)

The surgical system of claim 1, the fluorescence image extractor being configured to perform cross-channel subtraction to obtain the third fluorescence image.