# US 2018/0322666 A1 · Imaging system and methods displaying a fused multidimensional reconstructed image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0322666 A1
- Document ID: US20180322666A1
- Status: Published application
- Kind: A1
- Application number: 16/030,400
- Filed: 2018-07-09
- Published: 2018-11-08
- Earliest claimed: 2012-08-30
- Assignees: TrueVision Systems, Inc.
- Applicants: TrueVision Systems, Inc.
- Inventors: Ashok Burton Tripathi; George Charles Polchin; Yen Ting Ng
- Web page: https://patentgrep.com/patent/US20180322666A1

## CPC

- G16H 30/40: for processing medical images, e.g. editing
- A61B 5/0077: {Devices for viewing the surface of the body, e.g. camera, magnifying lens}
- A61B 5/7425: {Displaying combinations of multiple images regardless of image source, e.g. displaying a reference anatomical image with a live image}
- A61B 5/743: {Displaying an image simultaneously with additional graphical information, e.g. symbols, charts, function plots}
- A61B 5/745: {using a holographic display}
- A61B 6/03: Computed tomography [CT]
- A61B 6/032: {Transmission computed tomography [CT]}
- A61B 6/037: {Emission tomography}
- A61B 90/20: Surgical microscopes characterised by non-optical aspects
- A61B 90/37: {Surgical systems with images on a monitor during operation}
- G06T 5/70: Denoising; Smoothing
- G06T 7/0014: {using an image reference approach}
- G06T 7/337: {involving reference images or patches}
- G06T 12/30: Image post-processing, e.g. metal artefact correction
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G16H 20/40: relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
- G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS
- H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00)
- H04N 13/30: Image reproducers (optical systems for producing stereoscopic or other three-dimensional [3D] effects G02B30/00)
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/3983: {Reference marker arrangements for use with image guided surgery}
- A61B 2576/00: Medical imaging apparatus involving image processing or analysis (A61B1/00009, A61B6/52 and A61B8/52 take precedence)
- A61B 2576/026: for the brain
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/10088: Magnetic resonance imaging [MRI]
- G06T 2207/20104: Interactive definition of region of interest [ROI]
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30016: Brain
- G06T 2207/30196: Human being; Person
- G06T 2207/30204: Marker
- G06T 2210/41: Medical
- G06T 2210/62: Semi-transparency
- G06T 2219/028: Multiple view windows (top-side-front-sagittal-orthogonal)
- G06T 2219/2004: Aligning objects, relative positioning of parts
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A system, method, and apparatus for displaying a fused reconstructed image with a multidimensional image are disclosed. An example imaging system receives a selection corresponding to a portion of a displayed multidimensional visualization of a surgical site. At the selected portion of the multidimensional visualization, the imaging system displays a portion of at least one of a three-dimensional image or model which corresponds to the selected multidimensional visualization such that the displayed portion of the at least one of the three-dimensional image or model is fused with the displayed multidimensional visualization.

## Claims

### Claim 1

(canceled)

### Claim at position 2

2: An imaging system comprising:
at least one photosensor configured to record a visualization of a surgical site;
a visualization generation system configured to create a multidimensional visualization based on the recorded visualization of the surgical site;
an interface configured to receive image data of the surgical site from at least one of a medical device or a server that stores the image data from a medical device;
a processor communicatively coupled to a display device and a memory, the memory storing instructions, which when executed by the processor, cause the processor to:
create a three-dimensional image or model based on the image data,
display the multidimensional visualization of the surgical site,
receive a selection corresponding to a portion of the displayed multidimensional visualization of the surgical site,
select a portion of the three-dimensional image or model that corresponds to the selected portion of the multidimensional visualization,
combine the portion of the three-dimensional image or model with the multidimensional visualization, and
cause the display device to display the portion of the three-dimensional image or model in addition to the multidimensional visualization.

### Claim at position 3 (depends on 2)

3: The imaging system of claim 2, wherein the image data includes at least one of pre-operative image data, perioperative image data, or intra-operative image data.

### Claim at position 4 (depends on 2)

4: The imaging system of claim 2, wherein the memory includes additional instructions, which when executed by the processor, cause the processor to create the three-dimensional image or model using vector-based or feature based construction to create outlines of objects located within sequential pre-operative image slices of the image data.

### Claim at position 5 (depends on 2)

5: The imaging system of claim 2, wherein the memory includes additional instructions, which when executed by the processor, cause the processor to create the three-dimensional image or model based on volumetric information of the image data,
wherein the volumetric information includes values defining an image at each point in a coordinate plane.

### Claim at position 6 (depends on 5)

6: The imaging system of claim 5, wherein the values of the volumetric information include at least one of density values or intensity values.

### Claim at position 7 (depends on 6)

7: The imaging system of claim 6, wherein the memory includes additional instructions, which when executed by the processor, cause the processor to create the three-dimensional image or model by using the at least one of the density values or intensity values to define a tissue.

### Claim at position 8 (depends on 2)

8: The imaging system of claim 2, wherein the image data includes data from at least one of a computed tomography (“CT”) scan, a computed tomography angiography (“CT-A”) scan, a magnetic resonance imaging (“MRI”) scan, or a positron emission tomography (“PET”) scan.

### Claim at position 9 (depends on 2)

9: The imaging system of claim 2, wherein the memory includes additional instructions, which when executed by the processor, cause the processor to at least one of fuse, merge, blend, join, and integrate the portion of the three-dimensional image or model with the multidimensional visualization at the selected portion of the multidimensional visualization.

### Claim at position 10 (depends on 2)

10: The imaging system of claim 2, wherein the memory includes additional instructions, which when executed by the processor, cause the processor to select the portion of the three-dimensional image or model by:
determining at least one of an orientation and coordinates of the selected portion of the multidimensional visualization;
determining at least one of a corresponding orientation and coordinates of the three-dimensional image or model; and
selecting the portion of the three-dimensional image or model that is associated with the at least one of the corresponding orientation and coordinates.

### Claim at position 11 (depends on 2)

11: The imaging system of claim 2, wherein the memory includes additional instructions, which when executed by the processor, cause the processor to combine the portion of the three-dimensional image or model with the multidimensional visualization by:
identifying at least one of a feature, a structure, and an object inside of or within proximity of the selected portion of the multidimensional visualization;
identifying at least one of a corresponding feature, structure, and object within the three-dimensional image or model; and
aligning the portion of the three-dimensional image or model with the selected portion of the multidimensional visualization such that the at least one identified feature, structure, and object of the multidimensional visualization is aligned with the at least one corresponding identified feature, structure, and object of the three-dimensional image or model.

### Claim at position 12 (depends on 11)

12: The imaging system of claim 11, wherein at least one of the feature, structure, and object includes at least one of brain tissue, a blood vessel, a tumor, bone tissue, muscle tissue, and a marker.

### Claim at position 13

13: An imaging apparatus comprising:
a visualization generation system configured to create a multidimensional visualization based on a recorded visualization of a surgical site;
an interface configured to receive image data of the surgical site from at least one of a medical device or a server that stores the image data from a medical device;
a processor communicatively coupled to a display device and a memory, the memory storing instructions, which when executed by the processor, cause the processor to:
create a reconstructed image based on the image data,
display the multidimensional visualization of the surgical site,
receive a selection corresponding to a portion of the displayed multidimensional visualization of the surgical site,
select a portion of the reconstructed image that corresponds to the selected portion of the multidimensional visualization,
combine the portion of the reconstructed image with the multidimensional visualization, and
cause the display device to display the portion of the reconstructed image in addition to the multidimensional visualization.

### Claim at position 14 (depends on 13)

14: The imaging apparatus of claim 13, wherein the reconstructed image includes a three-dimensional image or model and the image data includes two-dimensional image slices.

### Claim at position 15 (depends on 14)

15: The imaging apparatus of claim 14, wherein the memory includes additional instructions, which when executed by the processor, cause the processor to create the three-dimensional image or model by:
determining a sequence of the two-dimensional image slices;
identifying a common structure between the image slices; and
forming corresponding three dimensional shapes based on the identified common structure.

### Claim at position 16 (depends on 13)

16: The imaging apparatus of claim 13, wherein the memory includes additional instructions, which when executed by the processor, cause the processor to:
create the reconstructed image at a first time before the visualization of the surgical site is recorded; and
display the multidimensional visualization of the surgical site at a second time after the visualization of the surgical site is recorded.

### Claim at position 17 (depends on 13)

17: The imaging apparatus of claim 13, wherein the memory includes additional instructions, which when executed by the processor, cause the processor to combine the portion of the reconstructed image with the multidimensional visualization by integrating, within a video signal, the portion of the reconstructed image with the multidimensional visualization at the selected portion.

### Claim at position 18 (depends on 13)

18: The imaging apparatus of claim 13, wherein the memory includes additional instructions, which when executed by the processor, cause the processor to combine the portion of the reconstructed image with the multidimensional visualization by adjusting a visual property of at least one of the multidimensional visualization and the portion of the reconstructed image so as to make the multidimensional visualization merge or fuse with the portion of the reconstructed image.

### Claim at position 19 (depends on 18)

19: The imaging apparatus of claim 18, wherein the memory includes additional instructions, which when executed by the processor, cause the processor to adjust the visual property by at least one of:
(a) increasing a transparency of the selected portion of a multidimensional visualization so that the portion of the reconstructed image replaces the selected portion of the multidimensional visualization;
(b) adjusting a contrast around edges or fringes of at least one of the selected portion of a multidimensional visualization and the portion of the reconstructed image;
(c) adjusting a focus around edges or fringes of at least one of the selected portion of a multidimensional visualization and the portion of the reconstructed image; and
(d) applying a spline function around edges or fringes of at least one of the selected portion of a multidimensional visualization and the portion of the reconstructed image.

### Claim at position 20 (depends on 13)

20: The imaging apparatus of claim 13, wherein the memory includes additional instructions, which when executed by the processor, cause the processor to combine the portion of the reconstructed image with the multidimensional visualization by:
identifying a target structure within the multidimensional visualization;
identifying the same target structure within the reconstructed image;
aligning the reconstructed image with the multidimensional visualization such that the target structure within the multidimensional visualization is aligned with the target structure within the reconstructed image; and
identifying the portion of the reconstructed image that is located at a same location as the selected portion of the multidimensional visualization.

### Claim at position 21 (depends on 20)

21: The imaging apparatus of claim 20, wherein the target structure includes a marked screw and is registered intra-operatively using an O-arm or C-arm imaging device.

### Claim at position 22 (depends on 13)

22: The imaging apparatus of claim 13, wherein the selected portion of the multidimensional visualization is represented as a window.

### Claim at position 23 (depends on 13)

23: The imaging apparatus of claim 13, wherein the memory includes additional instructions, which when executed by the processor, cause the processor to transmit an instruction to the display device specifying a location of the selected portion of the multidimensional visualization to which the portion of the reconstructed image is to be superimposed as a graphic.
