Vision-based scene segmentation and action classification

A method of training a video classifier, comprising:

• obtaining a plurality of videos depicting an object;
• training a first machine learning model to extract features associated with the object based on the plurality of videos;
• extracting a plurality of features from the plurality of videos using the trained first machine learning model;
• and
• training a second machine learning model to classify actions associated with the object in individual frames of video based on the plurality of features extracted by the first machine learning model.

A controller for a medical system, comprising:

• a processing system;
• a memory storing instructions that, when executed by the processing system, cause the controller to:
• obtain video captured by a camera disposed on a distal end of a medical instrument;
• detect a presence of an object in a series of frames of the video;
• extract features associated with the object from one or more frames in the series of frames using a first machine learning model;
• and
• infer an action associated with the object, for each frame of the one or more frames, using the features extracted from the frame and a second machine learning model.