Multi-modal medical image processing

#The present invention relates to the identification of regions of interest in medical images. More particularly, the present invention relates to the identification of regions of interest in medical images based on encoding and/or classification methods trained on multiple types of medical imaging data.

#Computer-aided diagnosis (also referred to as CAD) is a technology that has been developed and used for the last 40 years by medical professionals to assist in the interpretation of medical images generated using techniques such as magnetic resonance imaging, x-ray imaging and ultrasound imaging.

#Medical imaging techniques such as these produce a large amount of image data, which typically is processed manually by a medical professional, such as a radiologist, and, specifically, is analysed or evaluated by that medical professional in relation to the diagnosis being determined. Typical diagnoses, for example, relate to cancer screening and detection, such as in mammography.

#Embodiments will now be described, by way of example only and with reference to the accompanying drawings having like-reference numerals, in which:

#FIG. 1 illustrates a training process for a multimodal neural encoder;

#FIG. 2 illustrates a more detailed view of the encoder-decoder architecture;

#Referring to FIG. 1, a first embodiment will now describe the training of a multimodal neural encoder 1.

#In this embodiment, there is provided a method for training a multimodal neural encoder 1 which, in this embodiment, is operable to receive two or more separate Digital Imaging and Communications in Medicine (DICOM) files 5, 6. Each of the DICOM files 5, 6 is input to a DICOM decoder 10, 11. Each of the DICOM files 5, 6 can be from different types of sensor or imaging technique (or “modality”), so from a different source of information. One DICOM file 5, for example, can be from X-ray data, and the other DICOM file 6 can be from histological data. Similarly, any combination of X-ray data, histology data, MRI data, mammography data and genetic sequence data can be used as inputs.

#The input data can be of any modality and with any type of dimensionality. As an example, one input may be two-dimensional mammographic image and a second input may be one-dimensional genetic data.