# US 2022/0277477 A1 · Image-based guidance for navigating tubular networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0277477 A1
- Document ID: US20220277477A1
- Status: Published application
- Kind: A1
- Application number: 17/664,226
- Filed: 2022-05-20
- Published: 2022-09-01
- Earliest claimed: 2018-03-27
- Assignees: Siemens Healthcare GmbH
- Applicants: Siemens Healthcare GmbH
- Inventors: Sebastian Schafer; Randolph M. Setser; Maddalena Strumia
- Web page: https://patentgrep.com/patent/US20220277477A1

## CPC

- A61B 1/2676: {Bronchoscopes}
- A61B 1/00009: {of image signals during a use of endoscope}
- A61B 5/062: {using magnetic field}
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 34/25: {User interfaces for surgical systems}
- G06T 7/11: Region-based segmentation
- G06T 7/174: involving the use of two or more images
- G06T 7/74: {involving reference images or patches}
- G06T 12/20: Inverse problem, i.e. transformations from projection space into object space
- G06T 12/30: Image post-processing, e.g. metal artefact correction
- A61B 5/066: {Superposing sensor position on an image of the patient, e.g. obtained by ultrasound or x-ray imaging}
- A61B 2034/2051: {Electromagnetic tracking systems}
- A61B 2034/2057: {Details of tracking cameras}
- A61B 2034/2065: {Tracking using image or pattern recognition}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/10064: Fluorescence image
- G06T 2207/10068: Endoscopic image
- G06T 2207/10072: Tomographic images
- G06T 2207/10081: Computed x-ray tomography [CT]
- G06T 2207/10088: Magnetic resonance imaging [MRI]
- G06T 2207/10121: Fluoroscopy
- G06T 2207/30021: Catheter; Guide wire
- G06T 2207/30061: Lung
- G06T 2207/30172: Centreline of tubular or elongated structure
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems and methods for image-based guidance for facilitating navigation of tubular networks. A region of interest in three-dimensional image data may first be segmented. An endoscopic instrument may be detected in two-dimensional intraoperative image data of the region of interest. A centerline of the detected endoscopic instrument may then be determined. The endoscopic instrument and the centerline may be backprojected to generate a three-dimensional backprojected volume. A device path of the endoscopic instrument may be generated based at least in part on the three-dimensional backprojected volume and the three-dimensional image data.

## Claims

### Claim 1

An image-based guidance method, comprising:
segmenting, by a processor, a region of interest in three-dimensional (3D) image data to generate a segmented region of interest;
detecting, by the processor, a region of an endoscopic instrument in two-dimensional (2D) intraoperative image data of the region of interest and determining a centerline of the detected region of the endoscopic instrument;
performing, by the processor, backprojection of the endoscopic instrument and the centerline to generate a 3D backprojected volume including the backprojected endoscopic instrument and the backprojected centerline;
determining, by the processor, a device path of the endoscopic instrument based at least in part on distances from a centerline of the segmented region of interest to the backprojected centerline of the backprojected endoscopic instrument; and
generating, by the processor, a visualization of the device path from a starting point to a target point.

### Claim 2 (depends on 1)

The method of claim 1 further comprises acquiring the 3D image data of the region of interest, wherein the region of interest comprises at least a portion of a bronchial tree.

### Claim 3 (depends on 1)

The method of claim 1 further comprises acquiring the 2D intraoperative image data of the region of interest during a medical procedure.

### Claim 4 (depends on 3)

The method of claim 3 wherein acquiring the 2D intraoperative image data comprises acquiring one or more 2D fluoroscopic images.

### Claim 5 (depends on 1)

The method of claim 1 further comprises determining a centerline of the segmented region of interest.

### Claim 6 (depends on 1)

The method of claim 1 further comprises identifying, via a user interface, the starting point and the target point of the endoscopic instrument in the segmented region of interest.

### Claim 7 (depends on 1)

The method of claim 1 wherein determining, by the processor, the device path of the endoscopic instrument based at least in part on the distances from the centerline of the segmented region of interest to the backprojected centerline of the backprojected endoscopic instrument comprises:
determining one or more segment groupings, wherein each of the one or more segment groupings comprises a set of one or more segments that directionally connect a main segment of the segmented region of interest to a most distal segment of the segmented region of interest;
determining one or more penalty function values for the one or more segment groupings, wherein the one or more penalty function values are based on shortest distances between a centerline of a segment of the segmented region of interest and the backprojected centerline; and
selecting at least one of the one or more segment groupings with a smallest penalty function value, wherein the selected segment grouping forms at least a portion of the device path.

### Claim 8 (depends on 7)

The method of claim 7 wherein determining the one or more penalty function values comprises determining a normalized sum of the shortest distances.

### Claim 9 (depends on 8)

The method of claim 8 wherein determining the one or more penalty function values further comprises calculating a magnitude of a deformation field to be applied to the one or more segment groupings to contain the endoscopic instrument.

### Claim 10 (depends on 1)

The method of claim 1 further comprises acquiring electromagnetic (EM) tracked 3D positions of a tip of the endoscopic instrument corresponding to a sequence of the intraoperative 2D image data at different time points.

### Claim 11 (depends on 10)

The method of claim 10 wherein determining, by the processor, the device path of the endoscopic instrument based at least in part on the distances from the centerline of the segmented region of interest to the backprojected centerline of the backprojected endoscopic instrument comprises:
selecting an image from the sequence of the intraoperative 2D image data that corresponds to a backprojected endoscopic instrument tip with a smallest distance to a corresponding EM tracked 3D position;
translating the EM tracked 3D positions into a curve representation; and
updating the curve representation to minimize a difference between the curve representation and a backprojected plane of the endoscopic instrument derived from the selected image, wherein the updated curve representation forms at least a portion of the device path.

### Claim 12 (depends on 11)

The method of claim 11 wherein translating the EM tracked 3D positions into the curve representation comprises translating the EM tracked 3D positions into a basis spline.

### Claim 13 (depends on 11)

The method of claim 11 wherein updating the curve representation comprises modifying supporting points of the curve representation to satisfy an objective function.