# US 11,304,683 B2 · Biopsy workflow using multimodal imaging

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,304,683 B2
- Document ID: US11304683B2
- Status: Granted
- Kind: B2
- Application number: 16/570,859
- Filed: 2019-09-13
- Published: 2021-03-18
- Granted: 2022-04-19
- Earliest claimed: 2019-09-13
- Assignees: General Electric Company
- Applicants: General Electric Company
- Inventors: Jhimli Mitra; Thomas Kwok-Fah Foo; Desmond Teck Beng Yeo; David Martin Mills; Soumya Ghose; Michael John MacDonald
- Web page: https://patentgrep.com/patent/US11304683B2

## CPC

- A61B 8/5261: {combining images from different diagnostic modalities, e.g. ultrasound and X-ray}
- A61B 5/0035: {adapted for acquisition of images from more than one imaging mode, e.g. combining MRI and optical tomography}
- A61B 5/055: involving electronic [EMR] or nuclear [NMR] magnetic resonance, e.g. magnetic resonance imaging
- A61B 5/4381: {Prostate evaluation or disorder diagnosis}
- A61B 5/6823: {Trunk, e.g., chest, back, abdomen, hip}
- A61B 5/7267: {involving training the classification device}
- A61B 10/0241: {for prostate}
- G06T 7/12: Edge-based segmentation
- G06T 7/33: using feature-based methods
- G06T 7/344: {involving models}
- G16H 30/40: for processing medical images, e.g. editing
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G06T 2207/10088: Magnetic resonance imaging [MRI]
- G06T 2207/10136: 3D ultrasound image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30081: Prostate
- G06T 2207/30096: Tumor; Lesion
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

The subject matter discussed herein relates to multi-modal image alignment to facilitate biopsy procedures and post-biopsy procedures. In one such example, prostate structures (or other suitable anatomic features or structures) are automatically segmented in pre-biopsy MR and pre-biopsy ultrasound images. Thereafter, pre-biopsy MR and pre-biopsy ultrasound contours are aligned. To account for non-linear deformation of the imaged anatomic structure, a patient-specific transformation model is trained via deep learning based at least in part on the pre-biopsy ultrasound images. The pre-biopsy ultrasound images that are overlaid with the pre-biopsy MR contours and based off the deformable transformation model are then aligned with the biopsy ultrasound images. Such real-time alignment using multi-modality imaging techniques provides guidance during the biopsy and post-biopsy system.

## Claims

### Claim 1

A multi-modal imaging method to generate a patient-specific transformation model, comprising the steps of:
acquiring pre-biopsy magnetic resonance (MR) image data and pre-biopsy ultrasound image data of a patient;
segmenting an anatomic region of interest and one or more lesions in an MR volume generated from the pre-biopsy MR image data;
segmenting the anatomic region of interest in an ultrasound volume generated from ultrasound image data;
aligning contours of the segmented anatomic region of interest in the MR volume with corresponding contours of the anatomic region of interest in the ultrasound volume to generate a pre-biopsy volume in which the MR volume and the ultrasound volume are aligned and on which contours of the one or more lesions from the MR volume are aligned;
training a deep learning transformation model based at least in part on the pre-biopsy ultrasound image data to create a deformable transformation model specific to the patient;
wherein the pre-biopsy ultrasound image data comprise longitudinal volume frames; and
wherein a range of motion of the pre-biopsy longitudinal volume frames is captured by the deep learning transformation model.

### Claim 2 (depends on 1)

The method of claim 1, wherein the pre-biopsy MR image data comprises two-dimensional (2D) MR images reconstructed into a 3D representation and the pre-biopsy ultrasound image data comprises three-dimensional (3D) ultrasound images acquired over time.

### Claim 3 (depends on 1)

The method of claim 1, wherein the deep learning transformation model is specific to the anatomic region of interest.

### Claim 4 (depends on 3)

The method of claim 3, wherein the anatomic region of interest comprises a prostate.

### Claim 5 (depends on 1)

The method of claim 1, wherein the pre-biopsy MR image data and pre-biopsy ultrasound image data are acquired simultaneously.

### Claim 6 (depends on 5)

The method of claim 5, wherein the pre-biopsy ultrasound image data is acquired using an MR compatible ultrasound probe.

### Claim 7 (depends on 1)

The method of claim 1, further comprising the steps of:
acquiring one or more biopsy ultrasound images of a patient during a biopsy procedure;
performing a deformable alignment using the deformable transformation model to align each biopsy ultrasound image with the pre-biopsy volume, wherein the contours of the one or more lesions are previously associated with each biopsy ultrasound image; and
providing a real-time visualization of the contours of the one or more lesions on some or all of the biopsy ultrasound images.

### Claim 8

A multi-modal imaging system comprising:
a memory encoding processor-executable routines; and
a processing component configured to access the memory and execute the processor-executable routines, wherein the routines, when executed by the processing component, cause the processing component to perform actions comprising:
acquiring pre-biopsy magnetic resonance (MR) image data and pre-biopsy ultrasound image data of a patient;
segmenting an anatomic region of interest and one or more lesions in an MR volume generated from the pre-biopsy MR image data;
segmenting the anatomic region of interest in an ultrasound volume generated from the ultrasound image data;
aligning contours of the segmented anatomic region of interest in the MR volume with corresponding contours of the anatomic region of interest in the ultrasound volume to generate a pre-biopsy volume in which the MR volume and the ultrasound volume are aligned and on which contours of the one or more lesions from the MR volume are aligned;
training a deep learning transformation model based at least in part on the pre-biopsy ultrasound image data to create a deformable transformation model specific to the patient;
acquiring one or more biopsy ultrasound images of the patient during a biopsy procedure;
performing a deformable alignment using the deformable transformation model to align each biopsy ultrasound image with the pre-biopsy volume, wherein the contours of the one or more lesions are associated with each biopsy ultrasound image;
providing a real-time visualization of the contours of the one or more lesions on some or all of the biopsy ultrasound images;
wherein the pre-biopsy ultrasound image data comprise longitudinal volume frames; and
wherein a range of motion of the pre-biopsy longitudinal volume frames is captured by the deep learning transformation model.

### Claim 9 (depends on 8)

The method of claim 8, wherein the pre-biopsy MR image data comprises two-dimensional (2D) MR images reconstructed into a 3D representation and the pre-biopsy ultrasound image data comprises three-dimensional (3D) ultrasound images.

### Claim 10 (depends on 8)

The method of claim 8, wherein the deep learning transformation model is specific to the anatomic region of interest.

### Claim 11 (depends on 8)

The method of claim 8, wherein the pre-biopsy MR image data and pre-biopsy ultrasound image data are acquired simultaneously.

### Claim 12

A method to register a real-time image for a biopsy system, comprising the steps of:
acquiring one or more biopsy ultrasound images of a patient during a biopsy procedure;
performing a deformable alignment using a deformable transformation model to align each biopsy ultrasound image with a pre-biopsy volume, wherein the deformable transformation is specific to the patient and wherein the contours of one or more lesions are associated with each biopsy ultrasound image;
providing a real-time visualization of the contours of the one or more lesions on some or all of the biopsy ultrasound images;
wherein the deformable transformation model is generated using a deep learning transformation model trained at least in part based on pre-biopsy ultrasound image data;
wherein the pre-biopsy ultrasound image data comprise longitudinal volume frames; and
wherein a range of motion of the pre-biopsy longitudinal volume frames is captured by the deep learning transformation model.

### Claim 13 (depends on 12)

The method of claim 12, wherein the biopsy procedure comprises a trans-perineal biopsy procedure.

### Claim 14 (depends on 12)

The method of claim 12, wherein the deep learning transformation model is specific to the anatomy of interest.