System, method, and apparatus for tracking and labeling surgery tools

A system for processing video images of a surgical site in real-time, the system comprises a digital surgical microscope (DSM) comprising a DSM camera;

• a processor;
• and a memory device storing computer-executable instructions that, when executed by the processor, causes the processor to:
• receive, in real-time, by a computing device having a processor, image data of an image in a surgical video stream captured by a camera of the digital surgical microscope, wherein the surgical video stream shows at least one tool;
• determine the order of receipt of the image;
• for the first and every [(n+1)×i]th image:
• apply the image data to a first trained neural network model to determine a location for a bounding box around a distal end point of each of the at least one tool;
• generate processed image data comprising a bounding box around the distal end of the at least one tool;
• apply the processed image data to a second trained neural network model to track the distal end point of each of the at least one tool;
• initialize a trajectory for each bounding box and setting up a Kalman filter for each trajectory;
• calculate camera motion compensation for the image data received as compared to the image data of an immediate previous image;
• predict parameters of the bounding box of each of the at least one tool using the Kalman filter associated with the trajectory;
• correct the prediction of the Kalman filter using the calculated camera motion compensation;
• match the bounding box as generated in the processed image data and the bounding box as predicted by the Kalman filter;
• mark a trajectory as present if a bounding box as predicted by the Kalman filter is matched with a bounding box in the processed image data and updating the Kalman filter with the processed image data concerning the matched bounding box;
• mark a trajectory as lost if a bounding box as predicted by the Kalman filter is not matched with a bounding box in the processed image data;
• send the present trajectories and the initialized trajectories to a user interface module;
• and
• display the trajectories at a display using the user interface module;
• for every image not the first or [(n+1)×i]th image:
• calculate camera motion compensation for the image data received as compared to the image data of the immediate previous image;
• predict the next parameters of the bounding box of each of the at least one tool using a Kalman filter and the calculated camera motion compensation;
• send the predicted image data to the user interface;
• and
• display the predicted image at a display using the user interface module,
• wherein n is a predetermined integer and i starts at 1 and increases by 1 each time,
• wherein the processed image data comprises a location and a size of the bounding box, a label, and the bounding box confidence,
• wherein the parameters of the bounding box comprise a location, a size, a velocity, and a velocity of size change of the bounding box, and
• wherein the trajectory is a collection of digital image data concerning a bounding box collected over time.