# US 2025/0152298 A1 · Surgical system with multi-modality image display

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0152298 A1
- Document ID: US20250152298A1
- Status: Published application
- Kind: A1
- Application number: 19/025,822
- Filed: 2025-01-16
- Published: 2025-05-15
- Earliest claimed: 2016-11-11
- Assignees: Intuitive Surgical Operations, Inc.
- Applicants: Intuitive Surgical Operations, Inc.
- Inventors: Karen C. Brandao; Mahdi Azizian; Simon P. DiMaio; Brian D. Hoffman; Jonathan M. Sorger
- Web page: https://patentgrep.com/patent/US20250152298A1

## CPC

- A61B 90/361: {Image-producing devices, e.g. surgical cameras}
- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 34/37: Leader-follower robots (A61B34/35 takes precedence)
- A61B 90/00: Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B 2017/00216: {with eye tracking or head position tracking control}
- A61B 2034/101: {Computer-aided simulation of surgical operations}
- A61B 2034/254: {being adapted depending on the stage of the surgical procedure}
- A61B 2090/364: {Correlation of different images or relation of image positions in respect to the body}
- A61B 2090/371: {with simultaneous use of two cameras}
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)

## Abstract

A method is provided to produce a multi-modality image of a surgical scene comprising: capture light reflected from the surgical scene; producing first image information corresponding to a first modality image; producing second image information corresponding to a second modality image; selecting a portion of the second image modality based at least in part upon anatomical structure information included within the selected portion; and producing simultaneously within a display at least a portion of the first modality image of the surgical scene and the selected portion of the second modality image.

## Claims

### Claim 1

(canceled)

### Claim 2

A surgical system comprising:
a processor and memory coupled to the processor, the memory storing executable instructions to cause the surgical system to:
receive, via one or more user interfaces providing a selection of a plurality of image modalities, a first selection of a first image modality and a second selection of a second image modality, the plurality of image modalities comprising different types of image enhancement to image data;
produce the first image modality of a first portion of the surgical scene based at least on the first selection of the first modality image;
produce the second image modality of a second portion of the surgical scene based at least on the second selection of the second modality image; and
display simultaneously the first image modality of the first portion of the surgical scene within a region of a display and the second image modality of the second portion of the surgical scene within a different region of the display.

### Claim 3 (depends on 2)

The surgical system of claim 2, wherein the first image modality includes one of clear, Near Band Imaging, Raman spectroscopy, fluorescence and hyperspectral imaging; and wherein the second image modality includes a different one of Near Band Imaging, Raman spectroscopy, fluorescence and hyperspectral imaging.

### Claim 4 (depends on 2)

The surgical system of claim 2, wherein the different types of image enhancement comprises two or more of Near Band Imaging, Raman spectroscopy, fluorescence and hyperspectral imaging

### Claim 5 (depends on 2)

The surgical system of claim 2, further comprising executable instructions to cause the surgical system to receive, via one or more user interfaces providing a selection of formats for a multi-modal display, a format selection from a plurality of different formats.

### Claim 6 (depends on 5)

The surgical system of claim 5, further comprising executable instructions to cause the surgical system to use the format selection to display the first image modality of the first portion of the surgical scene within a region of the display and the second image modality of the second portion of the surgical scene within a different region of the display

### Claim 7 (depends on 2)

The surgical system of claim 2, wherein the format selection comprises one of a stitch, picture-in-picture or annotated format.

### Claim 8 (depends on 2)

The surgical system of claim 2, further comprising executable instructions to cause the surgical system to receive at least the first selection of the first modality image or the second selection of the second modality image responsive to eye tracking input.

### Claim 9

A surgical system comprising:
a processor and memory coupled to the processor, the memory storing executable instructions to cause the surgical system to:
receive surgical stage information;
determine that the surgical stage information matches one of a surgical stage rule or surgical stage signature that identifies a first image modality to use for displaying at least a first portion of a surgical scene;
produce a first portion of the surgical scene using the first image modality; and
display the first image modality of the first portion of the surgical scene within a region of a display.

### Claim 10 (depends on 9)

The surgical system of claim 9, wherein the surgical stage information comprises a system actuation state and surgical instrument kinematics.

### Claim 11 (depends on 9)

The surgical system of claim 9, wherein the surgical stage information comprises patient health record information.

### Claim 12 (depends on 9)

The surgical system of claim 9, wherein the surgical stage information comprises surgeon information.

### Claim 13 (depends on 9)

The surgical system of claim 9, further comprising executable instructions to cause the surgical system to display simultaneously with the first image modality of the first portion a second image modality of a second portion of the surgical scene within a different region of the display.

### Claim 14 (depends on 9)

The surgical system of claim 9, wherein the surgical stage signature comprises a multi-dimensional vector.

### Claim 15 (depends on 14)

The surgical system of claim 14, wherein the multi-dimensional vector comprises one or more values that are indicative of one or more attributes of a surgical stage.

### Claim 16 (depends on 14)

The surgical system of claim 14, wherein the multi-dimensional vector comprises one or more values indicative of location of one or more anatomical structures, location of one or more instruments, patient health, type of surgery, physician experience level and prior motion of an instrument.

### Claim 17

A surgical system comprising:
one or more eye sensors configured to provide eye tracking of a user;
a processor and memory coupled to the processor, the memory storing executable instructions to cause the surgical system to:
receive, via eye tracking input of the user, a first selection of a first image modality from a plurality of image modalities comprising different types of image enhancement to image data;
produce the first image modality of the first portion of the surgical scene based at least on the first selection of the first modality image; and
display the first image modality of the first portion of the surgical scene within a region of a display.

### Claim 18 (depends on 17)

The surgical system of claim 17, wherein the first image modality includes one of the different types of image enhancement comprising Near Band Imaging, Raman spectroscopy, fluorescence and hyperspectral imaging.

### Claim 19 (depends on 17)

The surgical system of claim 17, further comprising executable instructions to cause the surgical system to receive, via the eye tracking input of the user, a second selection of a second image modality from the plurality of image modalities.

### Claim 20 (depends on 17)

The surgical system of claim 17, further comprising executable instructions to cause the surgical system to produce the second image modality of the second portion of the surgical scene based at least on the second selection of the second image modality.

### Claim 21 (depends on 17)

The surgical system of claim 17, further comprising executable instructions to cause the surgical system to display simultaneously the first image modality of the first portion of the surgical scene within the region of the display and the second image modality of the second portion of the surgical scene within a different region of the display.
