Tracking multiple surgical tools in a surgical video

An apparatus, comprising:

• one or more processors;
• and
• a memory coupled to the one or more processors, wherein the memory stores instructions that, when executed by the one or more processors, cause the apparatus to:
• receive a current video frame of a surgical video for processing;
• receive one or more established tracks for one or more previously-detected surgical tools in the surgical video, wherein each established track comprises a first array that has a plurality of previously-predicted locations of a previously-detected surgical tool and a second array of semantic features associated with the previously-detected surgical tool across a plurality of previously-processed video frames of the surgical video;
• process the current video frame of the surgical video to detect one or more objects using a first deep-learning model;
• and
• for each detected object in the one or more detected objects,
• compute a semantic similarity metric between the detected object and each of the one or more established tracks by:
• using a second deep-learning model to extract a set of semantic features of the detected object;
• determining whether an established track comprises a detection gap in which a corresponding previously-detected surgical tool is temporarily out of view from a first subset of previously-processed video frames of the plurality of previously-processed video frames immediately before to the current video frame over a predefined time interval;
• and
• responsive to the established track comprising the detection gap, comparing the set of semantic features with one or more semantic features of the second array across a second subset of previously-processed video frames of the plurality of previously-processed video frames immediately before to the first subset of previously-processed video frames to determine whether the detected object and the corresponding previously-detected surgical tool are visually similar;
• compute a spatial similarity metric between the detected object and a latest predicted location in the first array for each of the one or more established tracks;
• and
• assign the detected object to one of the one or more established tracks based on the computed semantic similarity metric and the computed spatial similarity metric.

A system, comprising:

• one or more processors;
• and
• a memory coupled to the one or more processors, wherein the memory stores instructions that, when executed by the one or more processors, cause the system to:
• receive a current video frame of a surgical video for processing;
• receive one or more established tracks for one or more previously-detected surgical tools in the surgical video, wherein each established track comprises a first array that has a plurality of previously-predicted locations of a previously-detected surgical tool and a second array of semantic features associated with the previously-detected surgical tool across a plurality of previously-processed video frames of the surgical video;
• process the current video frame of the surgical video to detect one or more objects using a first deep-learning model;
• and
• for each detected object in the one or more detected objects,
• compute a semantic similarity metric between the detected object and each of the one or more established tracks by:
• using a second deep-learning model to extract a set of semantic features of the detected object;
• determining whether an established track comprises a detection gap in which a corresponding previously-detected surgical tool is temporarily out of view from a first subset of previously-processed video frames of the plurality of previously-processed video frames immediately before to the current video frame over a predefined time interval;
• and
• responsive to the established track comprising the detection gap, comparing the set of semantic features with one or more semantic features of the second array across a second subset of previously-processed video frames of the plurality of previously-processed video frames immediately before to the first subset of previously-processed video frames to determine whether the detected object and the corresponding previously-detected surgical tool are visually similar;
• compute a spatial similarity metric between the detected object and a latest predicted location in the first array for each of the one or more established tracks;
• and
• assign the detected object to one of the one or more established tracks based on the computed semantic similarity metric and the computed spatial similarity metric.