Modality-agnostic method for medical image representation

A computer-implemented method for processing imaging data using a trained modality-agnostic model, the method comprising:

• receiving medical imaging data of a first imaging modality;
• encoding the medical imaging data based on a defined data encoding, by mapping the medical imaging data to a latent representation;
• processing the encoded medical imaging data with a trained model, wherein the trained model produces an output based on the latent representation, based on training of the model to a fused latent representation of training imaging data, wherein training of the model is performed based on a mapping of a latent representation of the training imaging data obtained from at least a second imaging modality, wherein the training of the model is performed from:
• encoding of the training imaging data based on a defined data encoding for each modality of the training imaging data, the encoding performed from mapping of the training imaging data to respective latent representations corresponding to each modality, and producing respective latent variables from applying the defined data encoding to the training imaging data, wherein the respective latent variables correspond to a spatial representation of the respective latent representations;
• and
• fusing of the encoded training imaging data, the fusing performed from mapping of the respective latent representations to a fused latent representation of the encoded training imaging data, wherein the mapping uses the respective latent representations and the respective latent variables to determine the fused latent representation, and wherein the mapping combines the respective latent representations into the fused latent representation while conserving structure indicated by the respective latent variables;
• and
• performing an image processing operation on the medical imaging data based on the output produced by the trained model.

A system for operation of a modality-agnostic imaging processing model, the system comprising:

• processing circuitry comprising at least one processor;
• and
• a storage medium comprising instructions, which when executed by the at least one processor, cause the processor to:
• process imaging data produced from a medical imaging modality;
• encode the imaging data to a latent representation based on a defined data encoding, wherein one or more latent variables are produced from applying the defined data encoding to the imaging data, and wherein the one or more latent variables correspond to a spatial representation of the latent representation;
• perform mapping of the encoded imaging data to a common latent representation, wherein the mapping uses the latent representation and the one or more latent variables to determine the common latent representation, and wherein the mapping combines the latent representation into the common latent representation while conserving structure indicated by the one or more latent variables;
• train a model for medical imaging processing, using the common latent representation;
• and
• utilize the trained model to produce an output from subsequent medical imaging data, in connection with an image processing operation.