Computer system for detecting a tool location in surgical video frame by generating gaussian point on the tool location and predicting surgical actions
AssigneeCsats, Inc.
InventorBokai Zhang
Life of this patent
Estimating the term…
Abstract
Systems, methods, and instrumentalities are disclosed for computer vision-based surgical workflow recognition using natural language processing (NLP) techniques. Surgical video of surgical procedures may be processed and analyzed, for example, to achieve workflow recognition. Surgical phases may be determined based on the surgical video and segmented to generate an annotated video representation. The annotated video representation of the surgical video may provide information associated with the surgical procedure. For example, the annotated video representation may provide information on surgical phases, surgical events, surgical tool usage, and/or the like.
Claims18
A system for identifying surgical actions in a surgical video, the system comprising:
- one or more processors;
- a memory storing (a) instructions and (b) a dataset comprising one or more triplets representing surgical video data, wherein each respective triplet comprises instrument data, verb data, and anatomical target data;
- an object detection module that uses the one or more processors to detect a tool location in at least one frame of the surgical video data;
- an attention module that uses the one or more processors to generate an attention map centered on the tool location wherein the attention module introduces local attention to the tool location by generating a Gaussian point centered on the tool location;
- a feature extraction module that implements a two-dimensional convolutional neural network (2D CNN) to extract features from the dataset based on the attention map;
- and
- a prediction module that uses a multi-stage temporal convolutional network (MS-TCN) to recognize, based on the extracted features, one or more surgical actions.
A method for identifying surgical actions in a surgical video, the method comprising:
- obtaining a dataset of surgical video data;
- detecting a tool location of a surgical tool in at least one frame of the surgical video data;
- obtaining an attention map overlaid on the tool location;
- locating one or more bounding boxes of triplets representing areas of interest using a YOLOv5 (You Only Look Once) neural network;
- extracting features from the dataset based on the attention map;
- and
- recognizing, based on the extracted features, one or more surgical actions.
A system for identifying surgical actions in a surgical video, the system comprising:
- one or more processors;
- a memory storing surgical video data;
- an object detection module that uses the one or more processors to detect a tool location in at least one frame of the surgical video data, wherein the object detection module comprises a YOLOv5 (You Only Look Once) neural network to locate one or more bounding boxes of triplets representing areas of interest in the surgical video data;
- a feature extraction module that uses Gaussian attention two-dimensional convolutional neural network to extract features associated with the surgical video data based on the one or more bounding boxes;
- and
- a prediction module that uses a multi-stage temporal convolutional network (MS-TCN) to recognize, based on the extracted features, one or more tool-tissue interactions.
Dependencies are inferred from the claim text.