# US 2025/0148790 A1 · Position-aware temporal graph networks for surgical phase recognition on laparoscopic videos

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0148790 A1
- Document ID: US20250148790A1
- Status: Published application
- Kind: A1
- Application number: 18/684,292
- Filed: 2022-08-18
- Published: 2025-05-08
- Earliest claimed: 2021-08-19
- Applicants: Digital Surgery Limited
- Inventors: Abdolrahim Kadkhodamohammadi; Imanol Luengo Muntion; Danail V. Stoyanov
- Web page: https://patentgrep.com/patent/US20250148790A1

## CPC

- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 20/49: {Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes}
- G06V 10/62: relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
- G06V 10/82: using neural networks
- G06V 10/84: using probabilistic graphical models from image or video features, e.g. Markov models or Bayesian networks
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 20/47: {Detecting features for summarising video content}
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Data captured during a surgical procedure can include video streams, such as from a laparoscopic camera. Technical solutions are described to facilitate online surgical phase recognition from the captured video stream(s). Surgical phase recognition is key in developing context-aware supporting systems for surgeons and medical teams in general. The technical solutions describe taking temporal context in videos into account by precise modeling of temporal neighborhoods in a video.

## Claims

### Claim 1

A computer-implemented method comprising:
computing, by a processor, using an encoder machine learning model, a plurality of frame features respectively corresponding to a plurality of video frames from a surgical video, each frame feature being a latent representation of the corresponding video frame from the surgical video;
generating, by the processor, using a decoder machine learning model, a position-aware temporal graph data structure that comprises a plurality of nodes and a plurality of edges, wherein each node represents a respective frame feature and an edge between two nodes indicates a relative position of the two nodes;
aggregating, by the processor, an embedding at each node, the embedding at a first node is computed by applying an aggregation function to the embedding of each node connected to the first node;
generating, by the processor, phase labels for the nodes based on the embedding at each node; and
identifying, by the processor, one or more surgical phases in the surgical video based on the phase labels.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein a subset of the nodes is associated with a first phase based on each of the subset of the nodes having the same phase label.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising, storing, by the processor, information about the one or more surgical phases, the information identifying the video frames from the surgical video corresponding to the one or more surgical phases.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the surgical video is captured using a camera that is one from a group comprising an endoscopic camera, a laparoscopic camera, a portable camera, and a stationary camera.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the phase labels are generated using computer vision based on the latent representation.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, further comprising, generating a user interface that comprises a progress bar with a plurality of sections, each section representing a respective surgical phase from the one or more surgical phases.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein the progress bar is updated in real-time as the surgical video is being captured and processed.

### Claim 8 (depends on 6)

The computer-implemented method of claim 6, wherein each of the sections is depicted using a respective visual attribute.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein the visual attribute comprises at least one of a color, transparency, icon, pattern, and shape.

### Claim 10 (depends on 6)

The computer-implemented method of claim 6, wherein selecting a section causes a playback of the surgical video to navigate to a surgical phase corresponding to the section.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, wherein the decoder machine learning model is a graph neural network.

### Claim 12 (depends on 11)

The computer-implemented method of claim 11, wherein the graph neural network comprises:
a first block comprising a series of calibration layers;
a second block comprising a predetermined number of graph convolution layers; and
a third block comprising a classification head.

### Claim 13

A system comprising:
a machine learning system comprising:
an encoder that is trained to encode a plurality of video frames of a surgical video into a corresponding plurality of frame features; and
a temporal decoder that is trained to segment the surgical video into a plurality of surgical phases, each surgical phase comprising a subset of the plurality of video frames, wherein segmenting the surgical video by the temporal decoder comprises:
generating a position-aware temporal graph that comprises a plurality of nodes and a plurality of edges, each node represents a corresponding frame feature, and an edge between two nodes is associated with a time step between the video frames associated with the frame features corresponding to the two nodes;
aggregating, at each node, information from one or more adjacent nodes of the each node; and
identifying a surgical phase represented by each video frame based on the information aggregated at the each node.

### Claim 14 (depends on 13)

The system of claim 13, wherein the machine learning system further comprises outputting the surgical phases identified.

### Claim 15 (depends on 13)

The system of claim 13, wherein a surgical phase represented by each video frame is identified based on a latent representation of the video frame that is encoded into a frame feature.

### Claim 16 (depends on 13)

The system of claim 13, wherein the position-aware temporal graph is generated using a graph neural network.

### Claim 17

A computer program product comprising a memory device having computer-executable instructions stored thereon, which when executed by one or more processors cause the one or more processors to perform a method to autonomously identify surgical phases in a surgical video, the method comprising:
generating, using a machine learning system, a position-aware temporal graph to represent the surgical video, the position-aware temporal graph comprises a plurality of nodes and a plurality of edges, each node comprises a latent representation of a corresponding video frame from the surgical video, and an edge between two nodes is associated with a time step between the video frames corresponding to the two nodes;
for each layer of a graph neural network, aggregating, at each node, latent representations of adjacent nodes at a predefined time step associated with each layer, the graph neural network comprising a predetermined number of layers; and
identifying a surgical phase represented by each video frame based on the aggregated information at the each node.

### Claim 18 (depends on 17)

The computer program product of claim 17, wherein the each layer of the graph neural network is associated with a distinct predefined time step.

### Claim 19 (depends on 17)

The computer program product of claim 17, wherein the method further comprises storing a starting timepoint and an ending timepoint of the surgical phase based on a set of sequential video frames identified to represent the surgical phase.

### Claim 20 (depends on 17)

The computer program product of claim 17, wherein the surgical video is a real-time video stream or the surgical video is processed post-operatively.

### Claim 21

(canceled)
