Multi arm machine learning models with attention for lesion segmentation

A system comprising:

• one or more data processors;
• and
• a non-transitory computer readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform a set of actions including:
• accessing a plurality of three-dimensional magnetic resonance imaging (MRI) images, wherein each of the plurality of three-dimensional MRI images depicts a same volume of a brain of a subject;
• and a first three-dimensional MRI image was generated using a first type of MRI sequence that is different from a second type of MRI sequence used to generate a second three-dimensional MRI image;
• processing, for each three-dimensional MRI image of the plurality of three-dimensional MRI images, the three-dimensional MRI image using one or more corresponding encoder arms of a machine-learning model to generate an encoding of the three-dimensional MRI image, wherein the one or more corresponding encoder arms comprise a plurality of encoder models, wherein the plurality of encoder models comprises a first encoder model trained for the first type of MRI sequence and a second encoder model trained for the second type of MRI sequence, and wherein the first three-dimensional MRI image is processed by the first encoder model and the second three-dimensional MRI image is processed by the second encoder model;
• concatenating the encodings of the plurality of three-dimensional MRI images to generate a concatenated representation;
• and
• processing the concatenated representation using a decoder arm of the machine-learning model to generate a prediction that identifies one or more portions of the volume of the brain predicted to depict at least part of a lesion.

A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform a set of actions including:

• accessing a plurality of three-dimensional magnetic resonance imaging (MRI) images, wherein each of the plurality of three-dimensional MRI images depicts a same volume of a brain of a subject;
• and a first three-dimensional MRI image was generated using a first type of MRI sequence that is different from a second type of MRI sequence used to generate a second three-dimensional MRI image;
• processing, for each three-dimensional MRI image of the plurality of three-dimensional MRI images, the three-dimensional MRI image using one or more corresponding encoder arms of a machine-learning model to generate an encoding of the three-dimensional MRI image, wherein the one or more corresponding encoder arms comprise a plurality of encoder models, wherein the plurality of encoder models comprises a first encoder model trained for the first type of MRI sequence and a second encoder model trained for the second type of MRI sequence, and wherein the first three-dimensional MRI image is processed by the first encoder model and the second three-dimensional MRI image is processed by the second encoder model;
• concatenating the encodings of the plurality of three-dimensional MRI images to generate a concatenated representation;
• and
• processing the concatenated representation using a decoder arm of the machine-learning model to generate a prediction that identifies one or more portions of the volume of the brain predicted to depict at least part of a lesion.