Systems and methods for processing electronic images using deep foundation models

A method of training a foundation model to process digital medical images to infer metadata from those images, the method comprising:

• receiving a plurality of digital medical images;
• generating a plurality of image tokens from the digital medical images, the image tokens being fixed-sized patches;
• removing a subset of the plurality of image tokens from each of the digital medical images to generate a remaining plurality of image tokens from each of the digital medical images;
• encoding, using an encoder, the remaining plurality of image tokens from each of the digital medical images;
• adding a classification token to the encoded image tokens;
• appending masked tokens with position encodings to each respective encoded image token;
• and
• reconstructing, using a decoder, the image tokens, such that the image tokens align with original image pixel values.

A system for processing digital medical images to infer metadata from those images, the system comprising:

• at least one memory storing instructions;
• and
• at least one processor configured to execute the instructions to perform operations comprising:
• receiving a plurality of digital medical images;
• receiving a prompt, the prompt being a request for a specific type of metadata to be inferred from the plurality of digital medical images;
• determining, using a trained foundation model, at least one feature descriptor from the plurality of digital medical images based on the prompt;
• and
• providing for output the at least one feature descriptor for each of the plurality of digital medical images.