Machine-learning-based visual-haptic system for robotic surgical platforms

A computer-implemented method for generating a warning signal in real-time about interaction between a surgical tool and a tissue, the method comprising the following real-time video and image processing operations:

• receiving a video:
• processing the video to extract a plurality of video segments that depict a target tool-tissue interaction, wherein the target tool-tissue interaction involves exerting a force by a surgical tool on a tissue;
• and
• for each video segment of the plurality of video segments, annotating a plurality of video frames of the video segment with a set of force levels, wherein the plurality of video frames contain images of the target tool-tissue interaction and are annotated based on mapping an appearance of the target tool-tissue interaction in each of the plurality of video frames to a given force level in the set of force levels.

An apparatus for generating a warning signal in real-time about interaction between a surgical tool and a tissue, the apparatus comprising:

• one or more processors:
• a memory coupled to the one or more processors, wherein the memory stores instructions that, when executed by the one or more processors, cause the apparatus to perform the following real-time video and image processing operations:
• receiving a video, wherein the video is an actual surgical video of a surgery being performed:
• processing the video to extract a plurality of video segments that depict a target tool-tissue interaction, wherein the target tool-tissue interaction involves exerting a force by a surgical tool on a tissue;
• and
• for each video segment of the plurality of video segments, annotating a plurality of video frames of the video segment with a set of force levels, wherein the plurality of video frames contain images of the target tool-tissue interaction and are annotated based on mapping an appearance of the target tool-tissue interaction in each of the plurality of video frames to a given force level in the set of force levels.

A robotic surgical system, comprising:

• a surgical tool coupled to a robotic arm:
• an endoscope configured to capture a surgical video;
• and
• a processor configured to:
• receive the surgical video:
• process the surgical video to extract a plurality of video segments that depict a target tool-tissue interaction, wherein the target tool-tissue interaction involves exerting a force by the surgical tool on a tissue:
• and
• for each video segment of the plurality of video segments, annotate a plurality of video frames of the video segment with a set of force levels, wherein the plurality of video frames contain images of the target tool-tissue interaction and are annotated based on mapping an appearance of the target tool-tissue interaction in each of the plurality of video frames to a given force level in the set of force levels.