# US 2025/0082173 A1 · Machine-learning-based visual-haptic system for robotic surgical platforms

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0082173 A1
- Document ID: US20250082173A1
- Status: Published application
- Kind: A1
- Application number: 18/885,050
- Filed: 2024-09-13
- Published: 2025-03-13
- Earliest claimed: 2018-09-12
- Assignees: Verb Surgical Inc.
- Applicants: Verb Surgical Inc.
- Inventors: Jagadish Venkataraman; Denise Ann Miller
- Web page: https://patentgrep.com/patent/US20250082173A1

## CPC

- A61B 1/00006: {of control signals}
- A61B 1/000096: {using artificial intelligence}
- A61B 34/25: {User interfaces for surgical systems}
- A61B 34/35: for telesurgery
- A61B 34/37: Leader-follower robots (A61B34/35 takes precedence)
- A61B 34/74: {Manipulators with manual electric input means}
- A61B 34/76: {Manipulators having means for providing feel, e.g. force or tactile feedback}
- G06N 20/00: Machine learning
- G06T 7/0012: {Biomedical image inspection}
- G16H 20/40: relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
- G16H 40/63: for local operation
- A61B 1/04: combined with photographic or television appliances
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 90/361: {Image-producing devices, e.g. surgical cameras}
- A61B 2034/302: {specifically adapted for manipulations within body cavities, e.g. within abdominal or thoracic cavities}
- A61B 2034/305: {Details of wrist mechanisms at distal ends of robotic arms}
- A61B 2034/743: {Keyboards}
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

Embodiments described herein provide examples of a machine-learning-based visual-haptic system for constructing visual-haptic models for various interactions between surgical tools and tissues. In one aspect, a process for constructing a visual-haptic model is disclosed. This process can begin by receiving a set of training videos. The process then processes each training video in the set of training videos to extract one or more video segments that depict a target tool-tissue interaction from the training video, wherein the target tool-tissue interaction involves exerting a force by one or more surgical tools on a tissue. Next, for each video segment in the set of video segments, the process annotates each video image in the video segment with a set of force levels predefined for the target tool-tissue interaction. The process subsequently trains a machine-learning model using the annotated video images to obtain a trained machine-learning model for the target tool-tissue interaction.

## Claims

### Claim at position 1

-20. (canceled)

### Claim 21

A computer-implemented method for generating a warning signal in real-time about interaction between a surgical tool and a tissue, the method comprising the following real-time video and image processing operations:
receiving a video:
processing the video to extract a plurality of video segments that depict a target tool-tissue interaction, wherein the target tool-tissue interaction involves exerting a force by a surgical tool on a tissue; and
for each video segment of the plurality of video segments, annotating a plurality of video frames of the video segment with a set of force levels, wherein the plurality of video frames contain images of the target tool-tissue interaction and are annotated based on mapping an appearance of the target tool-tissue interaction in each of the plurality of video frames to a given force level in the set of force levels.

### Claim 22 (depends on 21)

The computer-implemented method of claim 21, wherein the video is an actual surgical video of a surgery being performed, and the set of force levels have been predefined for the target tool-tissue interaction.

### Claim 23 (depends on 21)

The computer-implemented method of claim 21, wherein mapping the appearance of the target tool-tissue interaction to the given force level comprises:
using the appearance to access a previously established set of visual-appearance standards for the target tool-tissue interaction, wherein each one of the previously established set of visual-appearance standards correlates a given appearance of the target tool-tissue interaction to a corresponding force level of the target tool-tissue interaction.

### Claim 24 (depends on 21)

The computer-implemented method of claim 21, further comprising:
prior to annotating the plurality of video frames, establishing the set of force levels for the target tool-tissue interaction by
establishing a set of visual-appearance standards for the target tool-tissue interaction wherein each visual-appearance standard in the set of visual-appearance standards correlates a given appearance of the target tool-tissue interaction to a corresponding level of the target tool-tissue interaction, and then
mapping the set of visual-appearance standards to the set of force levels, wherein the set of force levels are indicative of various degrees of the target tool-tissue interaction.

### Claim 25 (depends on 24)

The computer-implemented method of claim 24, wherein establishing a visual-appearance standard in the set of visual-appearance standards includes:
receiving a plurality of expert opinions from a plurality of experts, respectively, wherein each of the plurality of expert opinions specifies a strength value assigned to a given appearance of the target tool-tissue interaction; and
establishing the visual-appearance standard as an average strength value being an average of the plurality of expert opinions.

### Claim 26 (depends on 24)

The computer-implemented method of claim 24, wherein the target tool-tissue interaction is tying a surgical knot onto a tissue, and wherein the set of force levels are indicative of various degrees of tightness of the surgical knot.

### Claim 27 (depends on 26)

The computer-implemented method of claim 26, wherein the given appearance of the target tool-tissue interaction includes one or more of:
a shape of the surgical knot; and
a shape of the tissue onto which the surgical knot is tied.

### Claim 28 (depends on 24)

The computer-implemented method of claim 24, wherein the target tool-tissue interaction is pulling on a tissue with a grasper tool, and wherein the set of force levels are a set of tension levels applied on the tissue.

### Claim 29 (depends on 28)

The computer-implemented method of claim 28, wherein the given appearance of the target tool-tissue interaction is a curvature of an edge of the tissue that is under applied tension.

### Claim 30 (depends on 24)

The computer-implemented method of claim 24, wherein the target tool-tissue interaction is compressing a tissue with a stapler tool, and wherein the set of force levels are a set of compression levels applied on the tissue.

### Claim 31 (depends on 30)

The computer-implemented method of claim 30, wherein the given appearance of the target tool-tissue interaction is one or more of:
a shape of a plurality of jaws of the stapler tool while compressing the tissue; and
a shape of the tissue while compressed.

### Claim 32 (depends on 21)

The computer-implemented method of claim 21, wherein the set of force levels comprises:
a low level:
a moderate level: and
a high level.

### Claim 33 (depends on 32)

The computer-implemented method of claim 32, wherein the set of force levels further comprises:
a maximum-safe level representing a safety threshold for applying force to the tissue; and
at least one additional level above the maximum-safe level.

### Claim 34 (depends on 21)

The computer-implemented method of claim 21 further comprising:
determining a strength level of the target tool-tissue interaction by comparing i) the given force level obtained by said mapping and ii) a measurement from a pressure sensor integrated in the surgical tool, and in response generating a warning signal that an excessive pressure is detected.

### Claim 35 (depends on 21)

The computer-implemented method of claim 21 wherein the target tool-tissue interaction is compressing a tissue with a stapler and the method further comprises:
determining a timing for firing the stapler using i) pressure sensor data of measurements from a pressure sensor integrated in the surgical tool in combination with ii) the given force level obtained by said mapping.

### Claim 36

An apparatus for generating a warning signal in real-time about interaction between a surgical tool and a tissue, the apparatus comprising:
one or more processors:
a memory coupled to the one or more processors, wherein the memory stores instructions that, when executed by the one or more processors, cause the apparatus to perform the following real-time video and image processing operations:
receiving a video, wherein the video is an actual surgical video of a surgery being performed:
processing the video to extract a plurality of video segments that depict a target tool-tissue interaction, wherein the target tool-tissue interaction involves exerting a force by a surgical tool on a tissue; and
for each video segment of the plurality of video segments, annotating a plurality of video frames of the video segment with a set of force levels, wherein the plurality of video frames contain images of the target tool-tissue interaction and are annotated based on mapping an appearance of the target tool-tissue interaction in each of the plurality of video frames to a given force level in the set of force levels.

### Claim 37 (depends on 36)

The apparatus of claim 36, wherein the memory stores instructions that, when executed by the one or more processors, cause the apparatus to:
determine a strength level of the target tool-tissue interaction by comparing i) the given force level obtained by said mapping and ii) a measurement from a pressure sensor integrated at a tip of the surgical tool, and in response generating a warning signal that an excessive pressure is detected.

### Claim 38 (depends on 36)

The apparatus of claim 36, wherein the memory stores instructions that, when executed by the one or more processors, cause the apparatus to map the appearance of the target tool-tissue interaction to the given force level by using the appearance to access a previously established set of visual-appearance standards for the target tool-tissue interaction, wherein each one of the previously established set of visual-appearance standards correlates a given appearance of the target tool-tissue interaction to a corresponding force level of the target tool-tissue interaction.

### Claim 39

A robotic surgical system, comprising:
a surgical tool coupled to a robotic arm:
an endoscope configured to capture a surgical video; and
a processor configured to:
receive the surgical video:
process the surgical video to extract a plurality of video segments that depict a target tool-tissue interaction, wherein the target tool-tissue interaction involves exerting a force by the surgical tool on a tissue: and
for each video segment of the plurality of video segments, annotate a plurality of video frames of the video segment with a set of force levels, wherein the plurality of video frames contain images of the target tool-tissue interaction and are annotated based on mapping an appearance of the target tool-tissue interaction in each of the plurality of video frames to a given force level in the set of force levels.

### Claim 40 (depends on 39)

The robotic surgical system of claim 39, wherein the processor is configured to determine a strength level of the target tool-tissue interaction by comparing i) the given force level obtained by said mapping and ii) a measurement from a pressure sensor integrated in the surgical tool, and in response generate a warning signal that an excessive pressure is detected.
