Live surgical aid for brain tumor resection using augmented reality and deep learning

An augmented reality method, comprising:

• receiving video images of an anatomical region from a user's perspective;
• receiving a first 3D medical image comprising a 3D tumor image and a second 3D medical image comprising a tractogram;
• segmenting the first 3D medical image according to anatomically-defined boundaries;
• recognizing the anatomical region in the images of the anatomical region in real time using a trained neural network;
• generating an augmented reality representation of the segmented first 3D medical image and the second 3D medical image, from the perspective of the user, comprising the tractogram registered based on the video images;
• and
• presenting the augmented reality representation through an augmented reality display system.

A non-transitory computer readable medium storing instructions for controlling a programmable processor to perform an augmented reality method, comprising:

• instructions for segmenting a first 3D medical image comprising an image of a tumor according to anatomically-defined boundaries;
• instructions for recognizing the anatomical region in the images of the anatomical region in real time using a trained neural network;
• instructions for generating an augmented reality representation of the segmented first 3D medical image and a second 3D medical image comprising a tractogram from the perspective of the user, the tractogram being registered with the first 3D medical images based on the video images;
• and
• instructions for presenting the augmented reality representation through an augmented reality display system.