Surgical simulator providing labeled data

A surgical simulator for simulating a surgical scenario, the surgical simulator comprising:

• a display system adapted to show simulated surgical videos to a user of the surgical simulator;
• a user interface adapted to correlate a physical action of the user with a simulated action of the surgical simulator;
• a controller including one or more processors coupled to memory, the display system, and the user interface, wherein the memory stores instructions that when executed by the one or more processors cause the surgical simulator to perform operations including:
• generating the simulated surgical videos, each representative of a simulation of the surgical scenario;
• associating simulated ground truth data from the simulation with the simulated surgical videos, wherein the simulated ground truth data corresponds to context information of at least one of a surgical instrument, an anatomical region, a surgical task, or the action simulated by the surgical simulator;
• annotating features of the simulated surgical videos based, at least in part, on the simulated ground truth data for training a machine learning model;
• pre-training the machine learning model with the features annotated from the simulated surgical videos, wherein the features include at least one of a separation distance between the surgical instrument and the anatomical region, temporal boundaries of a surgical complication, or spatial boundaries of the surgical complication, and wherein the pre-training configures the machine learning model to probabilistically identify the features from unlabeled simulated videos corresponding to the surgical scenario;
• and
• after the pre-training, training the machine learning model with real surgical videos, wherein the training configures the machine learning model to probabilistically identify the features from unlabeled real videos corresponding to the surgical scenario.

A non-transitory computer-readable storage medium having stored thereon instructions which, when executed by one or more processing units, cause the one or more processing units to perform operations comprising:

• obtaining simulated surgical videos from a surgical video database, wherein each of the simulated surgical videos is representative of a simulation of a surgical scenario generated by a surgical simulator, wherein each of the simulated surgical videos includes simulated ground truth data corresponding to context information of at least one of a surgical instrument, an anatomical region, or a surgical task simulated by the surgical simulator;
• annotating features of the simulated surgical videos based, at least in part, on the simulated ground truth data for training a machine learning model;
• pre-training the machine learning model with the features annotated from the simulated surgical videos, wherein the features include at least one of a separation distance between the surgical instrument and the anatomical region, temporal boundaries of a surgical complication, or spatial boundaries of the surgical complication, wherein the pre-training configures the machine learning model to probabilistically identify the features from unlabeled videos corresponding to the surgical scenario;
• and
• saving the machine learning model and associated parameters configured by the pre-training to a data store included in the surgical simulator;
• and
• after the pre-training, training the machine learning model with real surgical videos, wherein the training configures the machine learning model to probabilistically identify the features from unlabeled real videos corresponding to the surgical scenario.