Augmented reality system with improved registration methods and methods for multi- therapeutic deliveries

A method for registration of anatomy of a patient using augmented reality for a procedure on a patient, comprising:

• providing a system including:
• an augmented reality system configured to display an augmented representation of an anatomical feature of the patient in an augmented reality;
• an imaging system configured to image the anatomical feature of the patient and generate an imaging dataset without the use of fiducial sensors;
• a tracked instrument configured to be used during the procedure and to generate a tracked instrument dataset;
• and
• a computer system in communication with the augmented reality system, the imaging system, and the tracked instrument;
• positioning the tracked instrument in a first axial plane and collecting, from the tracked instrument, the tracked instrument dataset;
• positioning the imaging system in a second axial plane and collecting, from the imaging system, the imaging dataset;
• identifying an internal landmark of the patient with both the tracked instrument and the imaging system;
• aligning the first axial plane and the second axial plane based on the internal landmark identified;
• generating, by the computer system, an imaging volume dataset based on the first axial plane and the second axial plane in 3D coordinates;
• registering, by the computer system, the imaging volume dataset with the augmented reality system;
• rendering the imaging volume dataset into an imaging volume hologram;
• projecting, by the augmented reality system, the imaging volume hologram;
• and
• registering a surgical robot for the procedure, including:
• attaching a robotic image target to a portion of the surgical robot at a predetermined position and a predetermined orientation;
• aligning the robotic image target within 3D cartesian coordinates;
• and
• localizing the robotic image target with augmented reality system coordinates;
• calibrating and registering the surgical robot for the procedure, including:
• programming the surgical robot to move the robotic image target in a predetermined pattern, wherein the predetermined pattern includes a plurality of predetermined periodic pauses;
• orienting the portion of the surgical robot that includes the robotic image target at each of the plurality of predetermined periodic pauses;
• capturing, by a camera of the augmented reality system, the robotic image target at each of the plurality of predetermined periodic pauses;
• corresponding each of the plurality of predetermined periodic pauses to a robotic coordinate;
• measuring, in a robotic coordinate system, a position of the robotic image target;
• and
• determining a transformation from the robotic coordinate system to the augmented reality system coordinates.