# US 2021/0243425 A1 · Optical imaging system and methods thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0243425 A1
- Document ID: US20210243425A1
- Status: Published application
- Kind: A1
- Application number: 17/238,737
- Filed: 2021-04-23
- Published: 2021-08-05
- Earliest claimed: 2015-02-06
- Assignees: The University of Akron
- Applicants: The University of Akron
- Inventors: Yang Liu
- Web page: https://patentgrep.com/patent/US20210243425A1

## CPC

- H04N 13/254: in combination with electromagnetic radiation sources for illuminating objects
- A61B 1/00006: {of control signals}
- A61B 1/045: Control thereof
- A61B 1/07: using light-conductive means, e.g. optical fibres
- A61B 5/0064: {Body surface scanning}
- A61B 5/0071: {by measuring fluorescence emission}
- A61B 6/032: {Transmission computed tomography [CT]}
- A61B 6/037: {Emission tomography}
- A61B 6/508: {for non-human patients}
- A61B 6/5247: {combining images from an ionising-radiation diagnostic technique and a non-ionising radiation diagnostic technique, e.g. X-ray and ultrasound}
- A61B 90/37: {Surgical systems with images on a monitor during operation}
- G06T 7/30: Determination of transform parameters for the alignment of images, i.e. image registration
- G06T 7/37: using transform domain methods
- G06V 20/64: Three-dimensional [3D] objects
- H04N 5/33: Transforming infrared radiation (cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20)
- H04N 13/246: Calibration of cameras
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- H04N 13/296: Synchronisation thereof; Control thereof
- H04N 23/21: from near infrared [NIR] radiation
- A61B 5/0075: {by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy (A61B5/0071 takes precedence)}
- A61B 5/7285: {for synchronizing or triggering a physiological measurement or image acquisition with a physiological event or waveform, e.g. an ECG signal}
- A61B 5/7425: {Displaying combinations of multiple images regardless of image source, e.g. displaying a reference anatomical image with a live image}
- A61B 5/7445: {Display arrangements, e.g. multiple display units}
- A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones}
- A61B 2090/373: {using light, e.g. by using optical scanners}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/10064: Fluorescence image
- G06T 2207/10072: Tomographic images
- G06T 2207/10076: 4D tomography; Time-sequential 3D tomography
- G06T 2207/10081: Computed x-ray tomography [CT]
- G06T 2207/10088: Magnetic resonance imaging [MRI]
- G06T 2207/10104: Positron emission tomography [PET]
- G06T 2207/10108: Single photon emission computed tomography [SPECT]
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An optical imaging system to image a target object includes a light source configured to emit one or more light rays to illuminate the target object and an image detector configured to capture a three-dimensional topography image of the target object when emitted light is emitted from the target object in response to being illuminated by the light rays emitted by the light source. A fluorescence image detector captures a fluorescence image of the target object when fluorescence is emitted from the target object in response illumination by light rays emitted by the light source. A controller instructs the image detector to capture the 3D topography image and the fluorescence image detector to detect the fluorescence image of the target object intraoperatively and to co-register and simultaneously display intraoperatively the co-registered topography and fluorescence information to the user via a display.

## Claims

### Claim 1

An optical imaging system to image a target object, comprising:
a light source that includes at least one projector and is configured to illuminate the target object with structured light; and
an image detector configured to capture a three-dimensional (3D) topography image of the target object when reflected light is reflected from the target object when illuminated by the structured light emitted by the light source;
a preoperative image detector configured to capture a preoperative image of the target object before the target object is positioned to be illuminated by the light source with the structured light;
a tracker that is configured to track a position of the target object; and
a controller that includes at least one graphics processing unit (GPU) and is configured to:
instruct the image detector to capture the 3D topography image of the target object intraoperatively,
co-register topography information detected from the 3D topography image intraoperatively and preoperative image information detected from the preoperative image detector to generate co-registered topography information and preoperative information, wherein the co-registered topography information and preoperative information is segmented to isolate an organ of interest and surface-based registration, and
display intraoperatively the co-registered topography information and the preoperative image information to the user via a display.

### Claim 2 (depends on 1)

The optical imaging system of claim 1, further comprising:
a computer topography (CT) detector configured to capture a CT scan image of the target object before the target object is positioned to be illuminated by the light source with light.

### Claim 3 (depends on 2)

The optical imaging system of claim 2, wherein the controller is further configured to co-register the topography information detected from the 3D topography image intraoperatively and CT scan image information detected from the CT detector to simultaneously display intraoperatively the co-registered topography information and the CT scan image information to the user via the display.

### Claim 4 (depends on 1)

The optical imaging system of claim 1, further comprising:
a magnetic resonance imaging (MRI) detector configured to capture an MRI scan image of the target object before the target object is positioned to be illuminated by the light emitter module with light.

### Claim 5 (depends on 4)

The optical imaging system of claim 4, wherein the controller is further configured to co-register the topography information detected from the 3D topography image intraoperatively and MRI scan image information detected from the MRI detector to simultaneously display intraoperatively the co-registered topography information and the CT scan image information to the user via the display.

### Claim 6 (depends on 1)

The optical imaging system of claim 1, further comprising:
a fluorescence detector configured to capture a fluorescence image of the target object when reflected light is reflected off the target object when illuminated by the light emitted by the light source.

### Claim 7 (depends on 6)

The optical imaging system of claim 6, wherein the controller is further configured to co-register the topography information detected from the 3D topography image intraoperatively and the preoperative information detected from the preoperative image detector and fluorescence information detected from the fluorescence image intraoperatively to simultaneously display intraoperatively the co-registered topography information, the preoperative information and the fluorescence information to the user via the display.

### Claim 8 (depends on 1)

The optical imaging system of claim 1, wherein the image detector is further configured to capture optical properties of biological tissue of the target object.

### Claim 9 (depends on 8)

The optical imaging system of claim 8, wherein the controller is further configured to incorporate optical properties associated with biological tissue of the target object to co-register the topography information detected from the 3D topography image intraoperatively and preoperative information detected from the preoperative image detector to generate the co-registered topography information and the preoperative image information.

### Claim 10 (depends on 1)

The optical imaging system of claim 1, wherein the controller is further configured to incorporate a three-dimensional (3D) surface profile from the 3D topography image to track tissue movements of the target object intraoperatively.

### Claim 11

An optical imaging system to image a target object, comprising:
a preoperative image detector configured to capture a preoperative image of the target object before the target object is positioned to be illuminated by at least one or more light rays projected onto the target object, wherein the preoperative image includes a three-dimensional (3D) topography of the target object;
an image projector configured to emit the one or more light rays to project a corrected projection image of the preoperative image of the target object to be visualized by a user;
a display configured to display the corrected projection image to be visualized by the user that is projected onto the display and enable the user to view via a surrounding environment of the user via the display simultaneously with the corrected projection image, wherein the display is partially transparent to light to enable the user to view the surrounding environment with natural vision simultaneously with the corrected projection image via the display;
a tracker that is configured to track a position of the target object; and
a controller that includes at least one graphics processing unit and is configured to:
map each relative distance determined from image information captured from the 3D topography of the target object included in the preoperative image to the corrected projection image based on a position of the image projector, wherein the corrected projection image incorporates each relative distance determined from the image information captured from the 3D topography of the target object included in the preoperative image to thereby display at least a portion of the 3D topography of the target object included in the preoperative image in the corrected projection image; and
instruct the image projector to project the corrected projection image to be visualized by the user via the display.

### Claim 12 (depends on 11)

The optical imaging system of claim 11, wherein the tracker is further configured to track the position of the target object based on optical tracking.

### Claim 13 (depends on 11)

The optical imaging system of claim 11, wherein the tracker is further configured to track the position of the target object based on electromagnetic tracking.

### Claim 14 (depends on 11)

The optical imaging system of claim 11, wherein the display is further configured to display the corrected projection image and enable the user to view the surrounding environment of the user simultaneously via at least one wearable display.

### Claim 15 (depends on 11)

The optical imaging system of claim 11, wherein the controller is further configured to:
calculate a transformation matrix between a preoperative image space that the corrected projection image is projected via the display and an intraoperative image space that an imaging system captures as the tracker tracks the position of the target object;
register preoperative image data included in the corrected projection image to the intraoperative image data included in the intraoperative space that is captured by the imaging system as the position of the target object is tracked by the tracker; and
instruct the display to display the preoperative image data included in the corrected projection image that is co-registered to the intraoperative image data that is captured by the imaging system as the position of the target object is tracked by the tracker.

### Claim 16

A method for imaging a target object, the method comprising steps of:
capturing by a preoperative image detector a preoperative image of the target object before the target object before the target object is positioned to be illuminated by at least one or more light rays projected onto the target object, wherein the preoperative image includes a three-dimensional (3D) topography of the target object;
emitting by an image projector the one or more light rays to project a corrected projection image of the preoperative image of the target object to be visualized by a user;
displaying by a display the corrected projection image to be visualized by the user that is projected onto the display and enable the user to view via a surrounding environment of the user via the display simultaneously with the corrected projection image, wherein the display is partially transparent to light to enable the user to view the surrounding environment with natural vision simultaneously with the corrected projection image via the display;
tracking by a tracker a position of the target object; and
mapping by a controller each relative distance determined from image information captured from the 3D topography of the target object included in the preoperative image to the corrected projection image based on a position of the image projector, wherein the corrected projection image incorporates each relative distance determined from the image information captured by the 3D topography of the target object included in the preoperative image to thereby display at least a portion of the 3D topography of the target object included in the preoperative image in the corrected projection image; and
instructing the image projector to project the corrected projection image to be visualized by the user via the display.

### Claim 17 (depends on 16)

The method of claim 16, wherein the step of tracking comprises tracking the position of the target object based on optical tracking.

### Claim 18 (depends on 16)

The method of claim 16, wherein the step of tracking further comprises tracking the position of the target object based on electromagnetic tracking.

### Claim 19 (depends on 16)

The method of claim 16, wherein the step of displaying comprises displaying the corrected projection image and enabling the user to view the surrounding environment of the user simultaneously via at least one wearable display.

### Claim 20 (depends on 16)

The method of claim 16, further comprising steps of:
calculating a transformation matrix between a preoperative image space that the corrected projection image is projected via the display and an intraoperative image space that an imaging system captures as the position of the target object is tracked;
registering preoperative image data included in the corrected projection image to the intraoperative image data included in the intraoperative space that is captured by the imaging system as the position of the target object is tracked by the tracker; and
instructing the display to display the preoperative image data included in the corrected projection image that is co-registered to the intraoperative image data that is captured by the imaging system as the position of the target object is tracked by the tracker.
