Generating multi-task frame predictions

A computer system comprising:

• a processor set;
• one or more computer-readable storage media;
• and
• program instructions stored on the one or more computer-readable storage media to cause the processor set to perform operations for generating multi-task frame predictions for a current frame of a video of a surgical procedure, the operations comprising:
• receiving historical video frames;
• generating a plurality of multi-task prompts based on the historical video frames;
• generating a plurality of spatial temporal embeddings based on the historical video frames and a current video frame;
• and
• generating the multi-task frame predictions based on the plurality of multi-task prompts and the plurality of spatial temporal embeddings.

A computer program product comprising:

• one or more computer-readable storage media;
• and
• program instructions stored on the one or more computer-readable storage media to perform operations for generating multi-task frame predictions for a current frame of a video of a surgical procedure, the operations comprising:
• receiving historical video frames;
• generating a plurality of multi-task prompts based on the historical video frames;
• generating a plurality of spatial temporal embeddings based on the historical video frames and a current video frame;
• and
• generating the multi-task frame predictions based on the plurality of multi-task prompts and the plurality of spatial temporal embeddings.