Explainability heat map for autonomous diagnostics based on high resolution medical image processing using vision transformer

A non-transitory computer-readable medium comprising memory with instructions encoded thereon for validating an autonomous diagnosis of a patient, the instructions when executed causing one or more processors to perform operations, the instructions comprising instructions to:

• receive a high-resolution image of a body part;
• divide the high-resolution image into a plurality of tiles;
• input a representation of each of the plurality of tiles into an encoder portion of a model, the model configured to perform a disease diagnosis based on the representations, and the encoder having an attention mechanism;
• obtain a plurality of tokens representative of an attention of the encoder based on the attention mechanism, each token associated with a position of a tile of the plurality of tiles;
• and
• generate for display a heat map corresponding to the image of the body part, the heat map comprising a two-dimensional image having pixels corresponding to tiles each with an amplitude based on a level of attention represented in the plurality of tokens corresponding to the tiles.

A method comprising:

• receiving a high-resolution image;
• dividing the high-resolution image into a plurality of tiles;
• inputting a representation of each of the plurality of tiles into an encoder portion of a model, the model having an attention mechanism;
• obtaining a plurality of tokens representative of an attention of the model based on the attention mechanism, each token associated with a position of a tile of the plurality of tiles;
• and
• generating for display a heat map, the heat map comprising a two-dimensional image having pixels corresponding to tiles each with an amplitude based on a level of attention represented in the plurality of tokens corresponding to the tiles.

A non-transitory computer-readable medium comprising memory with instructions encoded thereon that, when executed, cause one or more processors to perform operations, the instructions comprising instructions to:

• receive a high-resolution image;
• divide the high-resolution image into a plurality of tiles;
• input a representation of each of the plurality of tiles into an encoder portion of a model, the model having an attention mechanism;
• obtain a plurality of tokens representative of an attention of the model based on the attention mechanism, each token associated with a position of a tile of the plurality of tiles;
• and
• generate for display a heat map, the heat map comprising a two-dimensional image having pixels corresponding to tiles each with an amplitude based on a level of attention represented in the plurality of tokens corresponding to the tiles.