Deep-learning-based real-time remaining surgery duration (RSD) estimation

A system for continuously predicting in real-time a remaining surgical duration (RSD) of a live surgical session of a surgical procedure based on a real-time endoscope video of the live surgical session, the method comprising, the system comprising:

• one or more processors;
• and
• a memory coupled to the one or more processors, wherein the memory stores instructions that, when executed by the one or more processors, cause the system to:
• receive a current frame of the endoscope video at a current time of the live surgical session, wherein the current time is among a sequence of prediction time points for making continuous RSD predictions during the live surgical session;
• randomly sample N−1 additional frames of the endoscope video corresponding to the elapsed portion of the live surgical session between the beginning of the endoscope video corresponding to the beginning of the live surgical session and the current frame corresponding to the current time;
• combine the N−1 randomly sampled frames and the current frame in the temporal order to obtain a set of N frames;
• feed the set of N frames into a trained RSD machine-learning (ML) model for the surgical procedure;
• and
• output a current RSD prediction from the trained RSD ML model based on the set of N frames.