System and method for augmenting and synchronizing a virtual model with a physical model

An AR synchronization computer comprising:

• a first module for receiving synchronization and navigation data from a navigation system, for receiving data representative of a virtual model from a virtual model database, and for receiving tracking data indicative of the position and angle of view of an augmented reality head mounted display with respect to a physical frame of reference;
• a second module for registering a physical model, for generating a physical frame of reference with respect to the registered physical model based on the synchronization and navigation data, and for registering the augmented reality head mounted display with the navigation system using the synchronization and navigation data, thereby enabling tracking movement of the augmented reality head mounted display with respect to the physical model;
• a third module for anchoring the virtual model to the physical frame of reference;
• a fourth module for rendering a virtual image from the virtual model in real time based on the received tracking data responsive to determining that the physical model is within a field of view of the augmented reality head mounted display;
• and
• a fifth module for streaming the virtual image to the augmented reality head mounted display, thereby generating a synchronized and augmented realty view of the physical model.

A system for synchronizing and augmenting a virtual model with a physical model, the system comprising:

• an augmented reality head mounted display;
• a virtual model database comprising a virtual three-dimensional model representative of a patient anatomy;
• a navigation system configured to generate synchronization and navigation data;
• and
• an augmented reality synchronization computer comprising one or more processors, one or more computer-readable tangible storage devices, and program instructions stored on at least one of the one or more storage devices for execution by at least one of the one or more processors, the program instructions configured to:
• receive synchronization and navigation data from a navigation system and generate a physical frame of reference with respect to a registered physical model based on the synchronization and navigation data;
• register an augmented reality head mounted display with the navigation system using the synchronization and navigation data, thereby enabling tracking movement of the augmented reality head mounted display with respect to the physical model;
• receive data representative of a virtual model from a virtual model database;
• anchor the virtual model to the physical frame of reference;
• receive tracking data indicative of the position and angle of view of the augmented reality head mounted display with respect to the physical frame of reference;
• render a virtual image from the virtual model in real time based on the received tracking data responsive to determining that the physical model is within a field of view of the augmented reality head mounted display;
• and
• stream the virtual image to the augmented reality head mounted display, thereby generating a synchronized and augmented realty view of the physical model.