System and method for generating textual descriptions from medical images

A system configured to generate a textual description from a medical image, comprising:

• an imaging modality classification module configured to receive a first medical image generated with a first image modality, and further configured to determine that the first modality is a specific one of a plurality of different modalities, wherein the imaging modality classification module is trained to differentiate between the plurality of different modalities;
• an anatomy classification module configured to determine an anatomy represented in the medical image, wherein the anatomy classification module is trained to differentiate between different anatomy in medical images;
• an orchestrator module configured to determine, based at least on the determined first modality, which of a plurality of different text generation models to utilize to generate a textual description from the medical image, wherein each of the plurality of different text generation models is utilized for a different modality;
• a text generation module configured to generate, utilizing the identified text generation model, a textual description from the medical image, wherein the text generation module is trained to generate the textual description based at least in part on information from the medical image;
• and
• a user interface configured to report the generated textual description.

A non-transitory computer readable medium encoded with computer executable instructions, which when executed by a processor, cause the process to:

• receive a medical image generated with a first modality;
• determine that the first modality is a specific one of a plurality of different modalities;
• determine that the medical image comprises information about a specific portion of an anatomy;
• identify, based at least on the determined first modality, which of a plurality of different text generation models to utilize to generate a textual description from the medical image, wherein each of the plurality of different text generation models is utilized for a different modality;
• generate, utilizing the identified text generation model, a textual description from the medical image;
• and
• report the generated textual description.