Virtual interaction with instruments in augmented reality

A system comprising one or more processors, and a non-transitory computer-readable medium including one or more sequences of instructions that, when executed by the one or more processors, cause the system to perform operations comprising:

• generating within a unified three-dimensional (3D) coordinate space:
• (i) a 3D virtual medical model positioned according to a model pose;
• and
• (ii) at least one virtual object associated with a physical instrument, the physical instrument having a current instrument pose based at least on current coordinates of one or more fiducial markers disposed on the physical instrument, in the unified 3D coordinate space;
• rendering an Augmented Reality (AR) display that includes concurrent display of the 3D virtual medical model and the virtual object;
• detecting one or more physical gestures associated with the physical instrument;
• identifying a change related to the physical instrument pose, in the unified 3D coordinate space, due to at least one of the detected physical gestures associated with the physical instrument;
• and
• modifying the AR display according to at least one virtual interaction related to the virtual object that incorporates the change of the physical instrument pose, wherein modifying the AR display comprises:
• modifying the AR display by displaying a virtual trajectory of the physical instrument in relation to a target point at an internal anatomical location represented by the 3D virtual medical model.

A computer program product (“product”) comprising a non-transitory computer-readable medium having a computer-readable program code embodied therein to be executed by one or more processors, the program code including instructions to:

• generating within a unified three-dimensional (3D) coordinate space:
• (i) a 3D virtual medical model positioned according to a model pose;
• and
• (ii) at least one virtual object associated with a physical instrument, the physical instrument having a current instrument pose based at least on current coordinates of one or more fiducial markers disposed on the physical instrument, in the unified 3D coordinate space;
• rendering an Augmented Reality (AR) display that includes concurrent display of the 3D virtual medical model and the virtual object;
• detecting one or more physical gestures associated with the physical instrument;
• identifying a change related to the physical instrument pose, in the unified 3D coordinate space, due to at least one of the detected physical gestures associated with the physical instrument;
• and
• modifying the AR display according to at least one virtual interaction related to the virtual object that incorporates the change of the physical instrument pose, wherein modifying the AR display comprises:
• modifying the AR display by displaying a virtual trajectory of the physical instrument in relation to a target point at an internal anatomical location represented by the 3D virtual medical model.