# US 2021/0259788 A1 · Surgical system with augmented reality display

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0259788 A1
- Document ID: US20210259788A1
- Status: Published application
- Kind: A1
- Application number: 17/176,572
- Filed: 2021-02-16
- Published: 2021-08-26
- Earliest claimed: 2016-12-19
- Assignees: Ethicon, Inc.
- Applicants: Ethicon LLC
- Inventors: Jeffrey S. Swayze; Joshua Young; Geoffrey S. Strobl; Andrew Beckman
- Web page: https://patentgrep.com/patent/US20210259788A1

## CPC

- A61B 34/25: {User interfaces for surgical systems}
- A61B 5/0077: {Devices for viewing the surface of the body, e.g. camera, magnifying lens}
- A61B 5/7225: {Details of analogue processing, e.g. isolation amplifier, gain or sensitivity adjustment, filtering, baseline or drift compensation (input circuits for detecting, measuring, or recording bioelectric or biomagnetic signals A61B5/30; specific diagnostic methods using bioelectric or biomagnetic signals A61B5/316)}
- A61B 17/32: Surgical cutting instruments ({A61B18/042 takes precedence; suture cutters A61B17/0467;} implements for ligaturing and cutting {A61B17/122, A61B17/12; instruments for rupturing the amniotic membrane A61B17/4208; specially adapted knives for eye surgery A61F9/0133})
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 34/35: for telesurgery
- A61B 90/36: {Image-producing devices or illumination devices not otherwise provided for}
- A61B 90/361: {Image-producing devices, e.g. surgical cameras}
- A61B 90/37: {Surgical systems with images on a monitor during operation}
- A61B 90/39: {Markers, e.g. radio-opaque or breast lesions markers}
- G06T 7/73: using feature-based methods
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 10/273: {removing elements interfering with the pattern to be recognised}
- G06V 20/20: in augmented reality scenes
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 13/337: using polarisation multiplexing
- H04N 13/344: with head-mounted left-right displays
- A61B 5/743: {Displaying an image simultaneously with additional graphical information, e.g. symbols, charts, function plots}
- A61B 17/29: Forceps for use in minimally invasive surgery
- A61B 34/37: Leader-follower robots (A61B34/35 takes precedence)
- A61B 90/30: Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure
- A61B 2017/00017: {Electrical control of surgical instruments}
- A61B 2017/00061: {spectrum}
- A61B 2017/07214: {Stapler heads}
- A61B 2034/2065: {Tracking using image or pattern recognition}
- A61B 2034/256: {having a database of accessory information, e.g. including context sensitive help or scientific articles}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/367: {creating a 3D dataset from 2D images using position information}
- A61B 2090/371: {with simultaneous use of two cameras}
- A61B 2090/372: {Details of monitor hardware}
- A61B 2090/373: {using light, e.g. by using optical scanners}
- A61B 2090/3937: {Visible markers}
- A61B 2090/3983: {Reference marker arrangements for use with image guided surgery}
- A61B 2090/502: {Headgear, e.g. helmet, spectacles}
- G06T 2207/10024: Color image
- G06T 2207/30004: Biomedical image processing
- G06T 2207/30204: Marker
- G06T 2210/41: Medical
- G06V 2201/034: of medical instruments
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A surgical system includes a detector that includes an array of pixels configured to detect light reflected by a surgical device and generate a first signal. The first signal includes a first dataset representative of a visible image of the surgical device. The surgical system also includes a processor configured to receive the first signal and a second signal representative of one or more operating parameters of the surgical device. The processor is also configured to generate a modified image of the surgical device that includes information related to one or more operating parameters.

## Claims

### Claim 1

A surgical system comprising:
a detector, comprising an array of pixels configured to detect light reflected by a surgical environment that includes a surgical device and generate a first signal comprising a first dataset representative of a visible image of the surgical environment and the surgical device; and
a processor configured to:
receive the first signal and a second signal representative of one or more operating parameters of the surgical device,
generate a modified image of the surgical device that includes at least a portion of the surgical environment and the surgical device, and information related to the one or more operating parameters.

### Claim 2 (depends on 1)

The system of claim 1, further comprising a display device configured to display the modified image, wherein the display device includes at least one of a monitor and a display integrated into an accessory worn by a surgeon.

### Claim 3 (depends on 1)

The system of claim 1, wherein the information related to the one or more parameters is displayed adjacent to the surgical device in the modified image.

### Claim 4 (depends on 1)

The system of claim 1, wherein the information related to the one or more parameters is displayed on the surgical device in the modified image.

### Claim 5 (depends on 1)

The system of claim 1, wherein the modified image replaces one or more of at least a portion of the surgical device and at least a portion of the surgical field.

### Claim 6 (depends on 1)

The surgical system of claim 1, wherein the first dataset comprises values representing color of the light detected by one or more pixels of the first array of pixels.

### Claim 7 (depends on 1)

The system of claim 1, wherein the processor is configured to identify data representative of the surgical instrument from the first dataset to determine orientation of the surgical instruction based on position of the detector with respect to the surgical instrument.

### Claim 8 (depends on 1)

The system of claim 1, wherein the surgical device comprises one or more markers, each of the one or more markers configured to reflect a predetermined frequency of light.

### Claim 9 (depends on 8)

The system of claim 8, wherein the processor filters the first signal to determine orientation of the surgical device through an image recognition algorithm based on location of the one or more markers.

### Claim 10 (depends on 1)

The system of claim 1, wherein the surgical system is a robotic surgical system and the robotic surgical system comprises at least one robotic arm configured to hold and manipulate the surgical device.

### Claim 11 (depends on 1)

The system of claim 1, wherein the surgical device has a functionality including at least one of cutting, stapling, and energy delivery.

### Claim 12 (depends on 11)

The system of claim 11, wherein the one or more parameters includes at least one of articulation angle, shaft rotation angle, position of the knife, motion of the knife, tissue location, and reload information.

### Claim 13

A surgical system comprising:
a first detector, comprising a first array of pixels configured to detect light reflected by a surgical device within a surgical environment having one or more regions with a predetermined color, the first array of pixels further being configured to generate a first signal comprising a first dataset representative of a first visible image;
a second detector, comprising a second array of pixels configured to detect light reflected by the surgical environment and generate a second signal comprising a second dataset representative of a second visible image; and
a processor configured to:
receive the first and second signals,
identify, from the first and second datasets, data representative of the surgical environment that does not include the one or more regions of the surgical device having the predetermined color, and
generate a modified image of the surgical environment based on the identified data.

### Claim 14 (depends on 13)

The surgical system of claim 13, wherein the predetermined color is green.

### Claim 15 (depends on 13)

The surgical system of claim 13, wherein the first dataset comprises values representing color of the light detected by one or more pixels of the first array of pixels and the second dataset comprises values representing color of the light detected by one or more pixels of the second array of pixels.

### Claim 16 (depends on 13)

The surgical system of claim 13, wherein the modified image does not include the one or more regions of the surgical device having the predetermined color.

### Claim 17 (depends on 13)

The surgical system of claim 13, wherein the modified image replaces a portion of an image of a surgical device within the surgical environment with a portion of an image of tissue within the surgical environment.

### Claim 18 (depends on 17)

The surgical system of claim 17, wherein the image of tissue is an image of tissue underlying the replaced portion of the image of the surgical device.

### Claim 19 (depends on 13)

The system of claim 13, further comprising a display device configured to display the modified image.

### Claim 20

A surgical method comprising:
detecting light reflected by a surgical device;
generating a first signal comprising a first dataset representative of an image of the surgical device;
receiving the first signal and a second signal representative of one or more operating parameters of the surgical device; and
generating a modified image of the surgical device that includes information related to one or more operating parameters.
