Similar medical image search

A method of retrieving similar medical images to an input query image, comprising the steps of:

• creating a reference library in the form of a multitude of digital medical images, each of which is associated with image metadata;
• supplying the digital medical images to one or more machine learning models and representing the digital medical images as an embedding in the form of a projection of the digital medical images in a feature space having a plurality of axes, wherein the embedding is characterized by two aspects of a similarity ranking:
• (1) visual similarity, and (2) semantic similarity, such that neighboring images in the feature space are visually similar and semantic information is represented by the axes of the feature space;
• retrieving similar medical images for an input query image within a radius r within the feature space based on a projection of the input query image into the feature space;
• displaying, on a user interface, the similar medical images;
• and
• displaying, on the user interface, aggregate information about the similar medical images, wherein the aggregate information comprises frequency of diagnostic keywords among the similar medical images.

A system for facilitating searching within a medical image, comprising:

• a computer system configured as a search tool for receiving an input query from a user in the form of a portion of a larger medical image, the computer system including a machine learning pattern recognizer trained to find one or more additional portions of the larger medical image which is similar to the input query;
• a module in the computer system configured to
• highlight the additional portions of the larger medical image;
• and
• a user interface for entering the input query and display of both the input query and the one or more additional portions found by the computer system along with the highlighted additional portions of the larger medical image.