System and method for providing weakly-supervised online action segmentation

A system for providing weakly-supervised online action segmentation comprising:

• a memory storing instructions when executed by a processor cause the processor to:
• receive image data associated with multi-view videos of a procedure, wherein the procedure involves a plurality of atomic actions;
• analyze the image data using weakly-supervised action segmentation to identify each of the plurality of atomic actions by using an ordered sequence of action labels;
• train a neural network with data pertaining to the plurality of atomic actions based on the weakly-supervised action segmentation;
• and
• execute online action segmentation to label atomic actions that are occurring in real-time based on the plurality of atomic actions trained to the neural network, wherein at least one computing system is controlled to provide automation or feedback with respect to real-time atomic actions involved in completing the procedure based on the online action segmentation.

A non-transitory computer readable storage medium storing instruction that when executed by a computer, which includes a processor perform a method, the method comprising:

• receiving image data associated with multi-view videos of a procedure, wherein the procedure involves a plurality of atomic actions;
• analyzing the image data using weakly-supervised action segmentation to identify each of the plurality of atomic actions by using an ordered sequence of action labels;
• training a neural network with data pertaining to the plurality of atomic actions based on the weakly-supervised action segmentation;
• and
• executing online action segmentation to label atomic actions that are occurring in real-time based on the plurality of atomic actions trained to the neural network, wherein at least one computing system is controlled to provide automation or feedback with respect to real-time atomic actions involved in completing the procedure based on the online action segmentation.