Method and System for Model Based Fusion on Pre-Operative Computed Tomography and Intra-Operative Fluoroscopy Using Transesophageal Echocardiography

An apparatus for fusing pre-operative computed tomography (CT) image data with a fluoroscopic image acquired using a fluoroscopic image acquisition device based on an ultrasound image acquired using an ultrasound probe, comprising:

• means for detecting a 2D location of the ultrasound probe in the fluoroscopic image;
• means for estimating a 3D pose of the ultrasound probe based on the detected 2D location of the ultrasound probe in the fluoroscopic image;
• and
• means for mapping the ultrasound image to a 3D coordinate system of the fluoroscopic image acquisition device based on the estimated 3D pose of the ultrasound probe;
• means for detecting contours of an anatomical structure in the ultrasound image;
• means for calculating a transformation between the ultrasound image and a pre-operative CT image based on the contours detected in the ultrasound image and a patient-specific physiological model extracted from the pre-operative CT image;
• and
• means for determining a final mapping between the pre-operative CT image and the fluoroscopic image based on the transformation between the ultrasound image and the patient-specific physiological model extracted from the pre-operative CT image and the mapping of the ultrasound image to the 3D coordinate system of the fluoroscopic image acquisition device.

A non-transitory computer readable medium storing computer program instructions for fusing pre-operative computed tomography (CT) image data with a fluoroscopic image acquired using a fluoroscopic image acquisition device based on an ultrasound image acquired using an ultrasound probe, the computer program instructions when executed on a processor cause the processor to perform operations comprising:

• detecting a 2D location of the ultrasound probe in the fluoroscopic image;
• estimating a 3D pose of the ultrasound probe based on the detected 2D location of the ultrasound probe in the fluoroscopic image;
• and
• mapping the ultrasound image to a 3D coordinate system of the fluoroscopic image acquisition device based on the estimated 3D pose of the ultrasound probe;
• detecting contours of an anatomical structure in the ultrasound image;
• calculating a transformation between the ultrasound image and a pre-operative CT image based on the contours detected in the ultrasound image and a patient-specific physiological model extracted from the pre-operative CT image;
• and
• determining a final mapping between the pre-operative CT image and the fluoroscopic image based on the transformation between the ultrasound image and the patient-specific physiological model extracted from the pre-operative CT image and the mapping of the ultrasound image to the 3D coordinate system of the fluoroscopic image acquisition device.