AI-based automatic tool presence and workflow/phase/activity recognition

A method for automatically identifying surgical instruments used during a bronchoscopy procedure by a robotic system, the method comprising:

• performing, using a machine learning classifier, a first analysis of a bronchoscopy video of a patient site captured by a video capture device of the robotic system, the first analysis configured to categorize a medical instrument in the bronchoscopy video into a first tool class of a plurality of tool classes;
• identifying a set of possible instrument types for the medical instrument in the bronchoscopy video based on the first tool class and an identified phase of the bronchoscopy procedure, wherein the set of possible instrument types includes one or more phase-specific medical instruments associated with the first tool class and known to be employed for the identified phase;
• tracking a motion of the medical instrument in the bronchoscopy video;
• and
• selecting a type of instrument from the set of possible instrument types for the medical instrument based at least in part on the tracked motion.

A computerized system for automatically identifying surgical instruments used during a bronchoscopy procedure, the computerized system comprising:

• one or more processors configured to:
• perform, using a machine learning classifier, a first analysis of a bronchoscopy video of a patient site captured by a video capture device, the first analysis configured to categorize a medical instrument in the bronchoscopy video into a first tool class of a plurality of tool classes;
• identify a set of possible instrument types for the medical instrument in the bronchoscopy video based on the first tool class and an identified phase of the bronchoscopy procedure, wherein the set of possible instrument types includes one or more phase-specific medical instruments associated with the first tool class and known to be employed for the identified phase;
• track a motion of the medical instrument in the bronchoscopy video;
• and
• select a type of instrument from the set of possible instrument types for the medical instrument based at least in part on the tracked motion.