Systems and methods for the segmentation of multi-modal image data

A computer implemented method of automatic segmentation of at least one three dimensional (3D) anatomical region of interest (ROI) that delineates at least one predefined anatomical structure of a target individual, comprising:

• receiving a plurality of 3D images of a target individual, each 3D image of the plurality of 3D images including the at least one predefined anatomical structure;
• inputting the plurality of 3D images into one or more encoding-contracting components of a multi-modal neural network to compute a plurality of intermediate outputs for the plurality of 3D images, wherein each respective processing component of one or more processing components comprises one or more 3D convolutional filters, and the intermediate outputs comprises a plurality of features obtained as outcomes of the one or more 3D convolutional filters;
• inputting the plurality of features into a single decoding-expanding component of the multi-modal neural network;
• and
• computing an indication of a segmented at least one 3D ROI that delineates the at least one predefined anatomical structure by a single decoding-expanding component, wherein each stage of the single decoding-expansion component is concatenated by skips of a plurality of outputs generated by each corresponding stage of the encoding-contracting component.

A computer implemented method of training a multi-modal neural network for outputting an automatic segmentation of at least one three dimensional (3D) anatomical region of interest (ROI) that delineates at least one predefined anatomical structure of a target individual, comprising:

• receiving a plurality of training sets each including a plurality of registered 3D images created based on different imaging modalities and depicting at least one predefined anatomical structure;
• training a multi-modal neural network to compute at least one anatomical ROI that delineates the at least one predefined anatomical structure according to an input of a set of a plurality of registered 3D images based on the different imaging modalities,
• wherein each processing component of the multi-modal neural network independently computes a respective intermediate output of a plurality of intermediate outputs for the corresponding respective 3D image of the plurality of 3D images,
• wherein each respective processing component of the plurality of processing components comprises 3D convolutional filters, and the intermediate outputs comprises a plurality of features obtained as outcomes of the 3D convolutional filters,
• wherein the plurality of features are merged before at least one last convolutional layer of the multi-modal neural network.