System and method for processing endoscopy images in real time with object detection and annotation

A system for training at least one machine learning model for use with analyzing medical image data for a medical procedure and a speech-to-text conversion algorithm, wherein the system comprises:

• a non-transitory computer-readable medium having stored thereon program instructions for training the machine learning model;
• and
• at least one processor that, when executing the program instructions, is configured to:
• apply an encoder to at least one training image to generate at least one feature for a training object of interest (OOI) in the at least one training image;
• select a class for the training OOI by applying the at least one feature to the at least one machine learning model;
• reconstruct, using a decoder, a labeled training image by associating the at least one feature with the training image and the selected class with which to train the at least one machine learning model;
• train the speech-to-text conversion algorithm to identify matches between new audio data and ground truth text using a speech dataset comprising the ground truth text and audio data for the ground truth text, thereby generating at least one text string;
• match the at least one text string with the training image by comparing a timestamp associated with capture of the training image and a timestamp associated with reception of the new audio data, and
• overlay the training OOI and the at least one text string on an annotated image;
• in response to speech in the new audio data is recognized as a request for at least one reference image with OOIs that have been classified with the selected class, process text generated from the speech to determine that the speech includes a command requesting the at least one reference image, retrieve the at least one reference image from a data store, display the at least one reference image to allow a user to compare the training image with the at least one reference image to determine whether classification of the training OOI is correct, receive a subsequent input from the user via speech that either confirms or dismisses the classification of the training OOI, and update the at least one machine learning model based on the subsequent input.

A method for analyzing medical image data for a medical procedure, wherein the method comprises:

• receiving at least one image from a series of images;
• determining when there is at least one object of interest (OOI) in the at least one image and, when there is at least one OOI, determining a classification for the at least one OOI, where both determinations are performed using at least one machine learning model;
• displaying the at least one image and any determined OOIs to a user on a display using a bounding box during the medical procedure;
• receiving an input audio signal including speech from the user during the medical procedure and recognizing the speech;
• when the speech is recognized as a comment on the at least one image during the medical procedure, converting the speech into at least one text string using a speech-to-text conversion algorithm and a terminology correction algorithm;
• matching the at least one text string with the at least one image for which the speech from the user was provided, by matching the at least one text string with the at least one image by comparing a timestamp associated with capture of the at least one image and a timestamp associated with reception of the input audio signal;
• and
• generating at least one annotated image in which the at least one text string is linked to the corresponding at least one image during the medical procedure;
• wherein when the speech is recognized as a request for at least one reference image with OOIs that have been classified with the same classification as the at least one OOI during the medical procedure, the method further comprises:
• processing texted generated from the speech determine that the speech includes a command requesting the at least one reference image,
• retrieving the at least one reference image from a data store;
• displaying the at least one reference image to allow the user to compare the at least one image with the at least one reference image to determine whether the classification of the at least one OOI is correct;
• receiving a subsequent input from the user via speech that either confirms or dismisses the classification of the at least one OOI;
• and
• updating the at least one machine learning model based on the subsequent input.