# US 2026/0087791 A1 · Systems and methods for surgical data classification

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0087791 A1
- Document ID: US20260087791A1
- Status: Published application
- Kind: A1
- Application number: 19/405,115
- Filed: 2025-12-01
- Published: 2026-03-26
- Earliest claimed: 2020-11-22
- Assignees: Intuitive Surgical Operations, Inc.
- Applicants: Intuitive Surgical Operations, Inc.
- Inventors: Aneeq Zia; Kiran Bhattacharyya; Anthony Jarc
- Web page: https://patentgrep.com/patent/US20260087791A1

## CPC

- G06V 10/811: {the classifiers operating on different input data, e.g. multi-modal recognition}
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- A61B 34/37: Leader-follower robots (A61B34/35 takes precedence)
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/44: {Event detection}
- G16H 40/63: for local operation
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

Various of the disclosed embodiments are directed to computer-implemented systems and methods for recognizing surgical tasks from surgical data. In some embodiments an ensemble model configured to receive video data, kinematics data, and system event data from the surgical theater may be implemented. The ensemble model may implement modular streams for processing the data, facilitating predictions even when less than all the data types are available. In some embodiments, smoothing operations may help facilitate more accurate prediction results. Various of the embodiments may be employed in real-time during surgery, providing predictions at per-second intervals.

## Claims

### Claim 1

A method for determining a surgical task classification, the method comprising:
receiving, by one or more processors, a plurality of sets of surgical data for a surgical procedure, each set of surgical data corresponding to a respective modality type;
identifying, by the one or more processors, for each modality type, a type of machine learning model to use based at least on the respective modality type;
executing, by the one or more processors, for each modality type, the identified type of machine learning model on features derived from the corresponding set of surgical data, of the plurality sets of surgical data, to generate a classification result for each modality type; and
determining, by the one or more processors, a surgical task classification for the surgical procedure based at least on the classification result for each modality type.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining the surgical task classification for the surgical procedure comprises merging the classification results for each modality type using a fusion classifier or fusion logic.

### Claim 3 (depends on 1)

The method of claim 1, further comprising identifying, by the one or more processors, for the modality type of video, the type of machine learning model comprising a convolutional neural network and at least one sequential neural network layer.

### Claim 4 (depends on 1)

The method of claim 1, further comprising identifying, by the one or more processors, for the modality type of kinematics, the type of machine learning model comprising a one-dimensional convolutional neural network and at least one sequential neural network layer.

### Claim 5 (depends on 1)

The method of claim 1, further comprising identifying, by the one or more processors, for the modality type of system events, the type of machine learning model comprising an ensemble of base models and a fusion model.

### Claim 6 (depends on 1)

The method of claim 1, further comprising identifying, by the one or more processors, for the modality type of patient-side kinematics, the type of machine learning model comprising a one-dimensional convolutional neural network and at least one sequential neural network layer.

### Claim 7 (depends on 1)

The method of claim 1, further comprising deriving, by the one or more processors, for the modality type of video, features comprising one or more of pixel values, spatial features, and temporal features extracted from sequences of image frames.

### Claim 8 (depends on 1)

The method of claim 1, further comprising deriving, by the one or more processors, for the modality type of kinematics, features comprising one or more of time-series sensor values, statistical measures, and dimensionality-reduced representations.

### Claim 9 (depends on 1)

The method of claim 1, further comprising deriving, by the one or more processors, for the modality type of system events, features comprising one or more of event type indicators, event timestamps, and event parameters.

### Claim 10 (depends on 1)

The method of claim 1, further comprising deriving, by the one or more processors, for the modality type of patient-side kinematics, features comprising one or more of time-series sensor values, principal component analysis outputs, and multi-sensor data.

### Claim 11

A system for determining a surgical task classification, comprising:
at least one processor, coupled to memory and configured to:
receive a plurality of sets of surgical data for a surgical procedure, each set of surgical data corresponding to a respective modality type;
identify, for each modality type, a type of machine learning model to use based at least on the respective modality type;
execute, for each modality type, the identified type of machine learning model on features derived from the corresponding set of surgical data to generate a classification result for each modality type; and
determine a surgical task classification for the surgical procedure based at least on the classification result for each modality type.

### Claim 12 (depends on 11)

The system of claim 11, wherein the at least one processor is further configured to identify, for the modality type of video, the type of machine learning model comprising a convolutional neural network and at least one sequential neural network layer.

### Claim 13 (depends on 11)

The system of claim 11, wherein the at least one processor is further configured to identify, for the modality type of kinematics, the type of machine learning model comprising a one-dimensional convolutional neural network and at least one sequential neural network layer.

### Claim 14 (depends on 11)

The system of claim 11, wherein the at least one processor is further configured to identify, for the modality type of system events, the type of machine learning model comprising an ensemble of base models and a fusion model.

### Claim 15 (depends on 11)

The system of claim 11, wherein the at least one processor is further configured to derive, for the modality type of video, features comprising pixel values, spatial features, and temporal features extracted from sequences of image frames.

### Claim 16 (depends on 11)

The system of claim 11, wherein the at least one processor is further configured to derive, for the modality type of kinematics, features comprising time-series sensor values, statistical measures, and dimensionality-reduced representations.

### Claim 17 (depends on 11)

The system of claim 11, wherein the at least one processor is further configured to derive, for the modality type of system events, features comprising event type indicators, event timestamps, and event parameters.

### Claim 18

A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for determining a surgical task classification, the method comprising:
receiving a plurality of sets of surgical data for a surgical procedure, each set of surgical data corresponding to a respective modality type;
identifying, for each modality type, a type of machine learning model to use based at least on the respective modality type;
executing, for each modality type, the identified type of machine learning model on features derived from the corresponding set of surgical data to generate a classification result for each modality type; and
determining a surgical task classification for the surgical procedure based at least on the classification result for each modality type.

### Claim 19 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein the instructions further cause the one or more processors to identify, for a modality type of video, the type of machine learning model comprising a convolutional neural network and at least one sequential neural network layer, and to derive for the modality type of video, features comprising pixel values, spatial features, and temporal features extracted from sequences of image frames.

### Claim 20 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein the instructions further cause the one or more processors to identify, for a modality type of kinematics, the type of machine learning model comprising a one-dimensional convolutional neural network and at least one sequential neural network layer; and derive, for the modality type of kinematics, features comprising time-series sensor values, statistical measures, and dimensionality-reduced representations.
