Augmented reality guidance system for cardiac interventional surgery

A system for image guidance, comprising a memory storing processor executable instructions and one or more processors to:

• receive a first image data set comprising a rigid anatomical structure serving as a fiducial marker of a subject and an anatomical target comprising a target tissue;
• segment the first image data set to identify the rigid anatomical structure and the target tissue and to generate a three-dimensional (3D) model of the rigid anatomical structure and the target tissue;
• receive at least one fluoroscopic image comprising the rigid anatomical structure and a medical device comprising a radiopaque marker, the at least one fluoroscopic image being acquired or selected at a physiological phase of the subject that was present during acquisition of the first image data set;
• apply a segmentation engine comprising a first machine learning model to the at least one fluoroscopic image to generate a mask of the rigid anatomical structure in the at least one fluoroscopic image, the mask indicating a location of the rigid anatomical structure in the at least one fluoroscopic image;
• provide the mask and a projection of the 3D model to a registration engine comprising a second machine learning model trained on pairs of masks and projections with known transformation parameters, the registration engine providing a registration transformation that aligns the mask of the rigid anatomical structure with the projection of the 3D model, the registration transformation being refined by determining at least one of a rotation angle, a translation, or a scaling factor based on a projection of the radiopaque marker in the at least one fluoroscopic image;
• and
• generate a 3D rendering of the medical device registered with a geometry of the anatomical target comprising the target tissue for intra-procedural guidance.