Conjunction of 2D and 3D visualizations in augmented reality

A non-transitory computer readable medium comprising instructions which, when executed by a computer, cause the computer to:

• acquire image data which describes at least one slice image of a patient's anatomy reconstructed from a three-dimensional image dataset registered with the patient's actual anatomy;
• acquire movable object data which describes a spatial position of a movable object, particularly a handheld device or instrument, with respect to the patient's actual anatomy;
• acquire object of interest data which describes a model and a spatial position of at least one object of interest with respect to the three-dimensional image dataset;
• determine, based on the image data, the movable object data and the object of interest data, augmentation data which describes the augmentation of a field of view provided via an augmented reality device, including the at least one slice image and a three-dimensional visual representation of the at least one object of interest positionally aligned with the at least one slice image in a merged visualization,
• wherein a sectional plane of the at least one slice image is defined based on the spatial position of the movable object.

A medical system for providing an augmentation of a field of view via an augmented reality device, comprising:

• at least one processor executing instructions causing the at least one processor to:
• acquire image data which describes at least one slice image of a patient's anatomy reconstructed from a three-dimensional image dataset registered with the patient's actual anatomy;
• acquire movable object data which describes a spatial position of a movable object, particularly a handheld device or instrument, with respect to the patient's actual anatomy;
• acquire object of interest data which describes a model and a spatial position of at least one object of interest with respect to the three-dimensional image dataset;
• determine, based on the image data, the movable object data and the object of interest data, augmentation data which describes the augmentation of the field of view provided via the augmented reality device, including the at least one slice image and a three-dimensional visual representation of the at least one object of interest positionally aligned with the at least one slice image in a merged visualization,
• wherein a sectional plane of the at least one slice image is defined based on the spatial position of the movable object;
• at least one electronic data storage device storing the three-dimensional image dataset of the patient's anatomy or the spatial position and a three-dimensional model of the at least one object of interest;
• a medical tracking system configured to determine the spatial position of the movable object;
• and
• an augmented reality device, configured to augment the field of view as seen through the augmented reality device with digital image content;
• wherein the at least one processor is operably coupled to:
• the at least one electronic data storage device for acquiring, from the at least one data storage device, the three-dimensional image dataset of the patient's anatomy or the spatial position and a three-dimensional model of the at least one object of interest;
• the medical tracking system for acquiring, from the medical tracking system, the spatial position of the movable object;
• and
• the augmented reality device for issuing a control signal to the augmented reality device for controlling the operation of the augmented reality device on the basis of the augmentation data.