# US 2021/0196424 A1 · Visualization systems using structured light

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0196424 A1
- Document ID: US20210196424A1
- Status: Published application
- Kind: A1
- Application number: 16/729,744
- Filed: 2019-12-30
- Published: 2021-07-01
- Earliest claimed: 2019-12-30
- Assignees: Ethicon, Inc.
- Applicants: Ethicon LLC
- Inventors: Frederick E. Shelton, IV; Jason L. Harris
- Web page: https://patentgrep.com/patent/US20210196424A1

## CPC

- A61B 1/000094: {extracting biological structures}
- A61B 1/0005: {combining images e.g. side-by-side, superimposed or tiled}
- A61B 1/00055: {for alerting the user}
- A61B 1/00087: {Tools (A61B17/00234 takes precedence)}
- A61B 1/00126: {optical, e.g. for light supply cables}
- A61B 1/00149: {using articulated arms}
- A61B 1/00154: {using guiding arrangements for insertion}
- A61B 1/0016: {using motor drive units}
- A61B 1/00193: {adapted for stereoscopic vision}
- A61B 1/00194: {adapted for three-dimensional imaging}
- A61B 1/009: {with bending or curvature detection of the insertion part}
- A61B 1/018: for receiving instruments
- A61B 1/043: {for fluorescence imaging}
- A61B 1/046: {for infrared imaging}
- A61B 1/053: {being detachable}
- A61B 1/0605: {for spatially modulated illumination}
- A61B 1/0638: {providing two or more wavelengths}
- A61B 1/0676: {at distal tip of an endoscope}
- A61B 1/07: using light-conductive means, e.g. optical fibres
- A61B 1/3132: {for laparoscopy}
- A61B 34/25: {User interfaces for surgical systems}
- A61B 90/30: Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure
- A61B 90/361: {Image-producing devices, e.g. surgical cameras}
- A61B 90/37: {Surgical systems with images on a monitor during operation}
- G01B 11/25: by projecting a pattern, e.g. {one or more lines,} moiré fringes on the object (G01B11/255 takes precedence {; image analysis for depth or shape recovery G06T7/50})
- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G16H 20/40: relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
- G16H 30/40: for processing medical images, e.g. editing
- G16H 40/63: for local operation
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G16H 50/50: for simulation or modelling of medical disorders
- A61B 17/072: for applying a row of staples in a single action, e.g. the staples being applied simultaneously
- A61B 34/30: Surgical robots
- A61B 34/37: Leader-follower robots (A61B34/35 takes precedence)
- A61B 2017/00017: {Electrical control of surgical instruments}
- A61B 2017/00128: {related to intensity or progress of surgical action}
- A61B 2017/00809: {Lung operations}
- A61B 2034/101: {Computer-aided simulation of surgical operations}
- A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones}
- A61B 2034/2048: {using an accelerometer or inertia sensor}
- A61B 2034/2055: {Optical tracking systems}
- A61B 2034/2059: {Mechanical position encoders}
- A61B 2034/2065: {Tracking using image or pattern recognition}
- A61B 2090/304: {using chemi-luminescent materials}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/367: {creating a 3D dataset from 2D images using position information}
- A61B 2090/371: {with simultaneous use of two cameras}
- A61B 2090/373: {using light, e.g. by using optical scanners}
- A61B 2090/3941: {Photoluminescent markers}
- A61B 2090/3979: {infrared}
- G06T 2210/41: Medical
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

A visualization system including multiple light sources, an image sensor configured to detect imaging data from the multiple light sources, and a control circuit is disclosed. At least one of the light sources is configured to emit a pattern of structured light. The control circuit is configured to receive the imaging data from the image sensor, generate a three-dimensional digital representation of the anatomical structure from the pattern of structured light detected by the imaging data, obtain metadata from the imaging data, overlay the metadata on the three-dimensional digital representation, receive updated imaging data from the image sensor, and generate an updated three-dimensional digital representation of the anatomical structure based on the updated imaging data. The visualization system can be communicatively coupled to a situational awareness module configured to determine a surgical scenario based on input signals from multiple surgical devices.

## Claims

### Claim 1

A surgical visualization system, comprising:
a first light source configured to emit a first pattern of structured light at a first wavelength;
a second light source configured to emit a second pattern of structured light at a second wavelength, wherein the second wavelength is different than the first wavelength;
an image sensor configured to detect the first pattern of structured light and the second pattern of structured light on an anatomical structure; and
a control circuit configured to:
receive first imaging data from the image sensor, wherein the first imaging data is indicative of an outer surface contour of the anatomical structure from the first pattern of structured light detected by the image sensor;
receive second imaging data from the image sensor, wherein the second imaging data is indicative of a subsurface contour of the anatomical structure from the second pattern of structured light; and
generate a three-dimensional digital representation of the anatomical structure including the outer surface contour and the subsurface contour.

### Claim 2 (depends on 1)

The surgical visualization system of claim 1, wherein the control circuit is further configured to:
receive a signal from a situational awareness module; and
in response to the signal from the situational awareness module, update the three-dimensional digital representation of the anatomical structure.

### Claim 3 (depends on 1)

The surgical visualization system of claim 1, wherein the image sensor is configured to detect phase shift data for the first light source.

### Claim 4 (depends on 3)

The surgical visualization system of claim 3, wherein the image sensor is configured to detect phase shift data for the second light source.

### Claim 5 (depends on 1)

The surgical visualization system of claim 1, wherein the control circuit is further configured to:
obtain metadata from the first and second imaging data; and
overlay the metadata with the three-dimensional digital representation.

### Claim 6 (depends on 5)

The surgical visualization system of claim 5, wherein the metadata comprises at least one of tissue velocity, tissue distortion, tissue irregularity, tissue vascularization, or identification of an embedded structure within the anatomical structure.

### Claim 7 (depends on 5)

The surgical visualization system of claim 5, wherein the metadata is indicative of tissue compression along a staple line.

### Claim 8

A non-transitory computer-readable medium storing computer readable instructions, which when executed, cause a machine to:
detect, by an image sensor, a first pattern of structured light on an anatomical surface contour and a second pattern of structured light on a subsurface contour of the anatomical surface contour;
transmit, to a control circuit, first imaging data indicative of the first pattern of structured light and second imaging data indicative of the second pattern of structured light;
process, by the control circuit, the first imaging data and the second imaging data;
generate, by the control circuit a three-dimensional digital representation of an anatomical structure including the anatomical surface contour and the subsurface contour; and
transmit, to a monitor, the three-dimensional digital representation of the anatomical structure.

### Claim 9 (depends on 8)

The non-transitory computer-readable medium storing computer readable instructions of claim 8, which when executed, further cause the machine to:
update, by the control circuit, the three-dimensional digital representation of the anatomical structure in real time in response to changes to the first pattern of structured light and changes to the second pattern of structured light detected by the image sensor; and
transmit, to the monitor, the updated three-dimensional digital representation of the anatomical structure in real time.

### Claim 10

A surgical system, comprising:
a situational awareness module configured to determine a surgical scenario based on input signals from multiple surgical devices; and
a visualization system comprising a plurality of light sources, an image sensor configured to detect imaging data from the plurality of light sources, and a control circuit communicatively coupled to the situational awareness module, wherein at least one of the plurality of light sources is configured to emit a pattern of structured light onto an anatomical structure, and wherein the control circuit is configured to:
receive the imaging data from the image sensor;
generate a three-dimensional digital representation of the anatomical structure from the pattern of structured light detected by the imaging data;
obtain metadata from the imaging data;
overlay the metadata on the three-dimensional digital representation;
receive updated imaging data from the image sensor;
generate an updated three-dimensional digital representation of the anatomical structure based on the updated imaging data; and
in response to a surgical scenario determined by the situational awareness module, update the overlaid metadata on the updated three-dimensional digital representation of the anatomical structure.

### Claim 11 (depends on 10)

The surgical system of claim 10, wherein the plurality of light sources comprises a plurality of coherent light sources.

### Claim 12 (depends on 11)

The surgical system of claim 11, wherein the metadata comprises phase shift data for the plurality of coherent light sources.

### Claim 13 (depends on 11)

The surgical system of claim 11, wherein the plurality of coherent light sources are configured to emit the pattern of structured light.

### Claim 14 (depends on 13)

The surgical system of claim 13, wherein the pattern of structured light is emitted from the coherent light sources at a plurality of different wavelengths having different tissue penetration depths.

### Claim 15 (depends on 14)

The surgical system of claim 14, wherein the image sensor is configured to capture the pattern of structured light at the different tissue penetration depths, and wherein the three-dimensional digital representation is generated from a surface pattern of structured light and a sub-surface pattern of structured light.

### Claim 16 (depends on 15)

The surgical system of claim 15, wherein the overlaid metadata is indicative of a tissue irregularity.

### Claim 17 (depends on 16)

The surgical system of claim 16, wherein the tissue irregularity comprises a sub-surface irregularity.

### Claim 18 (depends on 10)

The surgical system of claim 10, wherein the overlaid metadata is indicative of tissue compression along a staple line.

### Claim 19 (depends on 10)

The surgical system of claim 10, wherein the overlaid metadata is indicative of a margin around an embedded structure.

### Claim 20 (depends on 10)

The surgical system of claim 10, wherein the overlaid metadata comprises at least one of tissue velocity, tissue distortion, tissue irregularity, tissue vascularization, and identification of an embedded structure within the anatomical structure.
