Anatomical Segmentation Identifying Modes and Viewpoints with Deep Learning Across Modalities

A computer program product comprising a non-transitory computer readable medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to implement a multi-modal classification and segmentation engine for anatomical segmentation identifying modes and viewpoints in biomedical images, wherein the computer readable program causes the computing device to:

• train a neural network perform simultaneous classification and segmentation using a set of training images, wherein the neural network provides a classification output that identifies a class label and a second output that identifies a segmentation label;
• provide, by the multi-modal classification and segmentation engine, a biomedical image as the input image to the neural network;
• output, by the neural network, a plurality of class label probabilities for a plurality of class labels and a plurality of segmentation label probabilities for each of a plurality of segmentation labels;
• classify, by a post-processing component executing within the multi-modal classification and segmentation engine, the biomedical image as an identified modality and an identified viewpoint based on the plurality of class label probabilities;
• segment, by the multi-modal classification and segmentation engine, the biomedical image based on the plurality of segmentation label probabilities;
• and
• output, by the multi-modal classification and segmentation engine, the classified and segmented biomedical image.

An apparatus, comprising:

• a processor;
• and
• a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to implement a multi-modal classification and segmentation engine for anatomical segmentation identifying modes and viewpoints in biomedical images, wherein the instructions cause the processor to:
• train a neural network perform simultaneous classification and segmentation using a set of training images, wherein the neural network provides a classification output that identifies a class label and a second output that identifies a segmentation label;
• provide, by the multi-modal classification and segmentation engine, a biomedical image as the input image to the neural network;
• output, by the neural network, a plurality of class label probabilities for a plurality of class labels and a plurality of segmentation label probabilities for each of a plurality of segmentation labels;
• classify, by a post-processing component executing within the multi-modal classification and segmentation engine, the biomedical image as an identified modality and an identified viewpoint based on the plurality of class label probabilities;
• segment, by the multi-modal classification and segmentation engine, the biomedical image based on the plurality of segmentation label probabilities;
• and
• output, by the multi-modal classification and segmentation engine, the classified and segmented biomedical image.