# US 12,118,738 B2 · Generation of three-dimensional scans for intraoperative imaging

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,738 B2
- Document ID: US12118738B2
- Status: Granted
- Kind: B2
- Application number: 18/358,396
- Filed: 2023-07-25
- Published: 2023-11-16
- Granted: 2024-10-15
- Earliest claimed: 2019-12-20
- Assignees: Unify Medical, Inc.
- Applicants: Unify Medical, Inc.
- Inventors: Yang Liu; Maziyar Askari Karchegani
- Web page: https://patentgrep.com/patent/US12118738B2

## CPC

- G06T 7/593: from stereo images
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 90/37: {Surgical systems with images on a monitor during operation}
- G06T 7/30: Determination of transform parameters for the alignment of images, i.e. image registration
- G06T 7/33: using feature-based methods
- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G06T 7/73: using feature-based methods
- H04N 13/106: Processing image signals (for multi-view video sequence encoding H04N19/597)
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- A61B 2034/2051: {Electromagnetic tracking systems}
- A61B 2034/2055: {Optical tracking systems}
- A61B 2034/2065: {Tracking using image or pattern recognition}
- A61B 2090/373: {using light, e.g. by using optical scanners}
- G06T 2207/10012: Stereo images
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30004: Biomedical image processing
- G06T 2207/30012: Spine; Backbone
- G06T 2207/30196: Human being; Person
- H04N 13/302: for viewing without the aid of special glasses, i.e. using autostereoscopic displays
- H04N 13/332: Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A system for executing a three-dimensional (3D) intraoperative scan of a patient is disclosed. A 3D scanner controller projects the object points included onto a first image plane and the object points onto a second image plane. The 3D scanner controller determines first epipolar lines associated with the first image plane and second epipolar lines associated with the second image plane based on an epipolar plane that triangulates the object points included in the first 2D intraoperative image to the object points included in the second 2D intraoperative image. Each epipolar lines provides a depth of each object as projected onto the first image plane and the second image plane. The 3D scanner controller converts the first 2D intraoperative image and the second 2D intraoperative image to the 3D intraoperative scan of the patient based on the depth of each object point provided by each corresponding epipolar line.

## Claims

### Claim 1

A system for executing a three-dimensional (3D) intraoperative scan of a patient to generate a plurality of intraoperative images of the patient that enables a surgeon to navigate during a surgical operation on the patient, comprising:
a 3D scanner that includes a projector, an image sensor, and a pattern generator, wherein:
the pattern generator is configured to generate a pseudo random pattern that includes a plurality of dots, wherein each position of each corresponding dot included in the pseuedo random pattern is pre-determined by the pattern generator,
the projector is configured to project the pseudo random pattern onto the patient, wherein each position of each corresponding dot included in the pseudo random pattern is projected on a corresponding position on the patient,
the image sensor is configured to capture a two-dimensional (2D) intraoperative image of a plurality of object points associated with the patient;
a 3D scanning controller that is configured to:
associate each object point associated with the patient that is captured by the image sensor with a corresponding dot included in the pseudo random pattern that is projected onto the patient by the projector based on the position of each corresponding dot as pre-determined by the pattern generator, and
convert the 2D intraoperative image to the 3D intraoperative scan of the patient based on the association of each object point to each position of each corresponding dot included in the pseudo random pattern as pre-determined by the pattern generator; and
a controller that is configured to:
co-register pre-operative image data captured from at least one pre-operative image of the patient with intraoperative image data provided by the 3D intraoperative scan, and
instruct a display to display the co-registered pre-operative image data as captured from the at least one pre-operative image with the intraoperative image data provided by the 3D intraoperative scan as the surgeon navigates during the surgical operation.

### Claim 2 (depends on 1)

The system of claim 1, wherein the projector is further configured to project the pseudo random pattern onto a 2D surface before the patient is positioned on the 2D surface.

### Claim 3 (depends on 2)

The system of claim 2, wherein the image sensor is further configured to capture a 2D image of the pseudo random pattern as projected onto the 2D surface before the patient is positioned on the 2D surface.

### Claim 4 (depends on 3)

The system of claim 3, wherein the 3D scanning controller is further configured to:
calibrate each position of each dot included in the pseudo random pattern as projected onto the 2D surface and pre-determined by the pattern generator to each corresponding position of each dot as included in the 2D image as captured by the image sensor;
compare each position of each dot included in the pseudo random pattern as projected onto the 2D surface and pre-determined by the pattern generator to each position of each dot included in the pseudo random pattern as projected onto the patient;
determine each depth of each object point as captured in the 2D intraoperative image by the image sensor of the patient when the projector projects the pseudo random pattern onto the patient after the calibration based on a difference in depth of each corresponding dot included in the pseudo random pattern as projected onto the 2D surface as compared to each corresponding dot included the pseudo random pattern as projected onto the patient; and
convert the 2D intraoperative image to the 3D intraoperative scan of the patient based on the depth of each object point as provided by the calibration of the pseudo random pattern to the 2D intraoperative image.

### Claim 5 (depends on 1)

The system of claim 1, wherein the 3D scanning controller is further configured to:
determine a plurality of first epipolar lines associated with a projection image plane of the projection of the pseudo random pattern and a plurality of second epipolar lines associated with associated with 2D intraoperative image plane of the captured 2D intraoperative image based on an epipolar plane that triangulates the plurality of object points included in the 2D intraoperative image to the plurality of dots included in the pseudo random pattern, wherein each epipolar line provides a depth of each object point as projected from the projection image plane associated with the projector and the 2D intraoperative image plane associated with the 2D intraoperative image; and
convert the 2D intraoperative image to the 3D intraoperative scan of the patient based on the depth of each object point provided by each corresponding epipolar line.

### Claim 6 (depends on 1)

The system of claim 1, wherein the 3D scanner and the 3D scanning controller is incorporated into a hand-held surgical navigation device.

### Claim 7 (depends on 1)

The system of claim 1, wherein the 3D scanning controller is further configured to change each pseudo random pattern projected onto the patient by the projector periodically to reduce residual pattern-to-depth dependence.