AI-assisted workflow segmentation

A method for automatically identifying phases of a medical procedure using a robotic system comprising a video capture device, a robotic manipulator, one or more sensors, and an input device, the method comprising:

• receiving sensor data from the one or more sensors;
• determining that the robotic manipulator is in a first position based on the sensor data from the one or more sensors;
• determining a set of procedure phases associated with the medical procedure;
• identifying a subset of procedure phases within the set of procedure phases based on at least one of the first position of the robotic manipulator or a user input;
• performing a first analysis of a video captured by the video capture device based on a machine learning algorithm configured to identify one or more features of the video;
• identifying a phase of the medical procedure from the set subset of procedure phases based at least in part on the one or more features of the video identified by the machine learning algorithm;
• and
• annotating the video with a first video marker associated with the identified phase of the medical procedure.

A control system for automatically identifying phases of a medical procedure performed by a robotic device, the control system comprising:

• a communication interface configured to receive sensor data indicating a position of a robotic device, user input data, and video data captured by a video capture device;
• memory configured to store the sensor data and the video data;
• and
• one or more processors configured to:
• determine that the robotic device is in a first position based on the sensor data from the one or more sensors;
• determine a set of procedure phases associated with the medical procedure;
• identify a subset of procedure phases within the set of procedure phases based on at least one of the first position of a robotic manipulator or a user input;
• perform an analysis of a video captured by a video capture device based on a machine learning algorithm configured to identify one or more features of the video;
• identify a phase of the medical procedure from the subset of procedure phases based at least in part on the one or more features of the video identified by the machine learning algorithm;
• and
• annotate the video with a first video marker associated with the identified phase of the medical procedure.