Machine-learning-based visual-haptic system for robotic surgical platforms

An apparatus for generating a haptic feedback based on a video containing interactions between surgical tools and their surrounding environment, the apparatus comprising:

• one or more processors;
• a memory coupled to the one or more processors, wherein the memory stores instructions that, when executed by the one or more processors, cause the apparatus to:
• receive a video of a surgical procedure;
• process the video to detect, in real time, a type of surgical tool interaction captured in the video;
• and
• apply a machine learning model to one or more video images in the video that are associated with the detected type of surgical tool interaction, to predict a target interaction between a surgical tool and its surrounding environment;
• and
• a user interface device including a mechanical component, wherein the processor configures the user interface device to output, via the mechanical component, the predicted target interaction to a surgeon who is performing the surgical procedure, as a haptic feedback signal.

A robotic system comprising:

• a surgical tool coupled to a robotic arm;
• an endoscope;
• one or more processors configured to:
• receive an endoscopic video captured by the endoscope of a surgical procedure;
• process the endoscopic video to detect a type of surgical tool interaction captured in the endoscopic video;
• and
• apply a machine learning model to one or more video images in the endoscopic video that are associated with the detected type of surgical tool interaction, to predict a target interaction between a surgical tool and its surrounding environment;
• and
• a user interface device including a mechanical component configured to output the predicted target interaction as a haptic feedback signal to a surgeon who is performing the surgical procedure.