Position-aware temporal graph networks for surgical phase recognition on laparoscopic videos
OwnerAbdolrahim Kadkhodamohammadi, Imanol Luengo Muntion +1inventors
InventorsAbdolrahim Kadkhodamohammadi, Imanol Luengo Muntion, Danail V. Stoyanov
Life of this patent
Estimating the term…
Abstract
Data captured during a surgical procedure can include video streams, such as from a laparoscopic camera. Technical solutions are described to facilitate online surgical phase recognition from the captured video stream(s). Surgical phase recognition is key in developing context-aware supporting systems for surgeons and medical teams in general. The technical solutions describe taking temporal context in videos into account by precise modeling of temporal neighborhoods in a video.
Claims21
A computer-implemented method comprising:
- computing, by a processor, using an encoder machine learning model, a plurality of frame features respectively corresponding to a plurality of video frames from a surgical video, each frame feature being a latent representation of the corresponding video frame from the surgical video;
- generating, by the processor, using a decoder machine learning model, a position-aware temporal graph data structure that comprises a plurality of nodes and a plurality of edges, wherein each node represents a respective frame feature and an edge between two nodes indicates a relative position of the two nodes;
- aggregating, by the processor, an embedding at each node, the embedding at a first node is computed by applying an aggregation function to the embedding of each node connected to the first node;
- generating, by the processor, phase labels for the nodes based on the embedding at each node;
- and
- identifying, by the processor, one or more surgical phases in the surgical video based on the phase labels.
A system comprising:
- a machine learning system comprising:
- an encoder that is trained to encode a plurality of video frames of a surgical video into a corresponding plurality of frame features;
- and
- a temporal decoder that is trained to segment the surgical video into a plurality of surgical phases, each surgical phase comprising a subset of the plurality of video frames, wherein segmenting the surgical video by the temporal decoder comprises:
- generating a position-aware temporal graph that comprises a plurality of nodes and a plurality of edges, each node represents a corresponding frame feature, and an edge between two nodes is associated with a time step between the video frames associated with the frame features corresponding to the two nodes;
- aggregating, at each node, information from one or more adjacent nodes of the each node;
- and
- identifying a surgical phase represented by each video frame based on the information aggregated at the each node.
A computer program product comprising a memory device having computer-executable instructions stored thereon, which when executed by one or more processors cause the one or more processors to perform a method to autonomously identify surgical phases in a surgical video, the method comprising:
- generating, using a machine learning system, a position-aware temporal graph to represent the surgical video, the position-aware temporal graph comprises a plurality of nodes and a plurality of edges, each node comprises a latent representation of a corresponding video frame from the surgical video, and an edge between two nodes is associated with a time step between the video frames corresponding to the two nodes;
- for each layer of a graph neural network, aggregating, at each node, latent representations of adjacent nodes at a predefined time step associated with each layer, the graph neural network comprising a predetermined number of layers;
- and
- identifying a surgical phase represented by each video frame based on the aggregated information at the each node.
(canceled)
Dependencies are inferred from the claim text.