# US 2025/0152297 A1 · Surgical system with multi-modality image display

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0152297 A1
- Document ID: US20250152297A1
- Status: Published application
- Kind: A1
- Application number: 19/025,799
- Filed: 2025-01-16
- Published: 2025-05-15
- Earliest claimed: 2016-11-11
- Assignees: Intuitive Surgical Operations, Inc.
- Applicants: Intuitive Surgical Operations, Inc.
- Inventors: Karen C. Brandao; Mahdi Azizian; Simon P. DiMaio; Brian D. Hoffman; Jonathan M. Sorger
- Web page: https://patentgrep.com/patent/US20250152297A1

## CPC

- A61B 90/361: {Image-producing devices, e.g. surgical cameras}
- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 34/37: Leader-follower robots (A61B34/35 takes precedence)
- A61B 90/00: Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B 2017/00216: {with eye tracking or head position tracking control}
- A61B 2034/101: {Computer-aided simulation of surgical operations}
- A61B 2034/254: {being adapted depending on the stage of the surgical procedure}
- A61B 2090/364: {Correlation of different images or relation of image positions in respect to the body}
- A61B 2090/371: {with simultaneous use of two cameras}
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)

## Abstract

A method is provided to produce a multi-modality image of a surgical scene comprising: capture light reflected from the surgical scene; producing first image information corresponding to a first modality image; producing second image information corresponding to a second modality image; selecting a portion of the second image modality based at least in part upon anatomical structure information included within the selected portion; and producing simultaneously within a display at least a portion of the first modality image of the surgical scene and the selected portion of the second modality image.

## Claims

### Claim 1

(canceled)

### Claim 2

A surgical system comprising:
a processor and memory coupled to the processor, the memory storing executable instructions to cause the surgical system to:
produce a first image modality of a surgical scene that provides a first type of enhancement to image data received by one or more cameras;
produce a second image modality of the surgical scene that provides a second type of enhancement to image data received by the one or more cameras;
stitch together a first portion of the first image modality and a second portion of the second image modality to produce a stitch format; and
display the stitch format in a selected region of a display.

### Claim 3 (depends on 2)

The surgical system of claim 2, further comprising executable instructions to cause the surgical system to select the first portion of the first image modality to stitch together with the second portion of the second image modality.

### Claim 4 (depends on 2)

The surgical system of claim 2, further comprising executable instructions to cause the surgical system to select, based at least on one or more rules, the first portion of the first image modality to stitch together with the second portion of the second image modality.

### Claim 5 (depends on 2)

The surgical system of claim 2, further comprising executable instructions to cause the surgical system to select, based at least on one or more anatomical structures within the image data, the first portion of the first image modality to stitch together with the second portion of the second image modality.

### Claim 6 (depends on 2)

The surgical system of claim 2, wherein the first image modality includes one of clear, Near Band Imaging, Raman spectroscopy, fluorescence, hyperspectral imaging; and wherein the second image modality includes a different one of Near Band Imaging, Raman spectroscopy, fluorescence and hyperspectral imaging.

### Claim 7 (depends on 2)

The surgical system of claim 2, wherein the first portion of the first image modality is stitched together with the second portion of the second image modality to produce a single surgical scene having different portions represented using different display modalities.

### Claim 8 (depends on 3)

The surgical system of claim 3, wherein the stitch format uses different image modalities to represent different portions of an image of a surgical scene such that only one image modality is visible at a given time within the stitched region of the image.

### Claim 9

A surgical system comprising:
a processor and memory coupled to the processor, the memory storing executable instructions to cause the surgical system to:
produce a first image modality of a surgical scene that provides a first type of enhancement to image data received by one or more cameras;
produce a second image modality of the surgical scene that provides a second type of enhancement to image data received by the one or more cameras;
select a second portion of the second modality to display at a reduced size within a first portion of the first image modality; and
display the first image modality of the surgical scene within a region of a display and the selected portion of the second image modality of the surgical scene at the reduced size within the first portion of the first image modality

### Claim 10 (depends on 9)

The surgical system of claim 9, wherein the first image modality includes one of clear, Near Band Imaging, Raman spectroscopy, fluorescence and hyperspectral imaging; and wherein the second image modality includes a different one of Near Band Imaging, Raman spectroscopy, fluorescence and hyperspectral imaging.

### Claim 11 (depends on 9)

The surgical system of claim 9, further comprising executable instructions to cause the surgical system to display the selected portion of the second image modality as a picture-in-picture of the first image modality.

### Claim 12 (depends on 9)

The surgical system of claim 9, further comprising executable instructions to cause the surgical system to display the selected portion of the second image using a picture-in-picture (PIP) format that displays a first video image of the surgical scene using the first display modality and display at the reduced size one of an insert video or image of the surgical scene using the second display modality.

### Claim 13 (depends on 9)

The surgical system of claim 9, further comprising executable instructions to cause the surgical system to select the selected portion of the second image based at least on an image signature.

### Claim 14 (depends on 9)

The surgical system of claim 9, further comprising executable instructions to cause the surgical system to receive a selection of a presentation style of a picture-in-picture (PIP) format from a user.

### Claim 15

A surgical system comprising:
a processor and memory coupled to the processor and storing executable instructions to cause the surgical system to:
receive surgical stage information;
produce a first image modality of a surgical scene from image data received by one or more cameras, the first image modality selected from one of a plurality types of image enhancements;
produce a second image modality of the surgical scene from image data received by one or more cameras, the second image modality selected from a different one of the plurality types of image enhancements; and
display simultaneously at least the first image modality of a first portion of the surgical scene within a region of a display and the second image modality of a second portion the surgical scene within a different region of the display.

### Claim 16 (depends on 15)

The system of claim 15, wherein the plurality of types of image enhancements includes clear, Near Band Imaging, Raman spectroscopy, fluorescence and hyperspectral imaging.

### Claim 17 (depends on 15)

The system of claim 15, further comprising executable instructions to cause the surgical system to display a modality selection user interface from which to select an image modality from the plurality of types of image enhancements.

### Claim 18 (depends on 15)

The system of claim 15, further comprising executable instructions to cause the surgical system to select a multi-modality display format based at least upon the first and second image modalities.

### Claim 19 (depends on 16)

The system of claim 16, wherein the multi-modality display format comprises one of a stitch, picture-in-picture or annotated format.

### Claim 20 (depends on 15)

The system of claim 15, wherein the first portion of the first image modality is stitched together with the second portion of the second image modality to produce a single surgical scene having different portions represented using different display modalities.

### Claim 21 (depends on 15)

The system of claim 15, further comprising executable instructions to cause the surgical system to display the second portion of the second image modality as a picture-in-picture within the first image modality.
