System and method to capture spatio-temporal representation for video reconstruction and analysis

A processor implemented method for segmentation and classification of a plurality of surgical tools used in a surgery, comprising:

• receiving, via one or more hardware processors, a video of the surgery as an input data, wherein the video of the surgery comprising a plurality of video frames;
• splitting, via the one or more hardware processors, the received video to the plurality of video frames of fixed length, wherein the plurality of video frames of fixed length is in a predefined sequence;
• fine-tuning, via the one or more hardware processors, pre-generated data model for segmentation of the plurality of surgical tools by:
• freezing one or more model weights of each layer of the encoder network and predefined layers of the decoder network;
• adding one or more layers of the 3D convolution unit and a softmax unit to the neural network, to map spatio-temporal representation to tool segmentation masks for each of the plurality of input video frames;
• and
• updating one or more model weights of unfrozen layers and the added one or more layers during training, to generate a fine-tuned data model;
• generating, via the one or more hardware processors, segmentation mask for each of the plurality of the video frames and for each of the plurality of surgical tools using the fine-tuned data model;
• training, via the one or more hardware processors, a neural network to classify the plurality of surgical tools by using:
• one or more 2D convolution unit layers;
• and
• one or more fully connected unit layers that make use of the generated segmentation mask to classify the plurality of surgical tools in each of the plurality of the video frames;
• and
• classifying, via the one or more hardware processors, each of the plurality of surgical tools using the trained neural network.

A system (100) for video analysis to capture a spatio-temporal representation comprising:

• one or more hardware processors (102);
• one or more communication interfaces (103);
• and
• a memory (101), wherein the memory comprises a plurality of instructions, which when executed, cause the one or more hardware processors to:
• receive a video as an input data, wherein the received video comprising a plurality of video frames;
• split the video to the plurality of video frames of fixed length, wherein the plurality of video frames of fixed length is in a predefined sequence;
• capture a spatial information for each of the plurality of video frames, by processing the predefined sequence of video frames, using a two-Dimensional (2D) convolution network;
• capture an optical flow information for each of the plurality of video frames;
• capture spatio-temporal representation of the plurality of video frames using an encoder, wherein the encoder comprising:
• a 3D convolution unit network to capture a first set of spatio-temporal features by processing the predefined sequence of the plurality of video frames, wherein the captured first set of spatio-temporal features is of a first-time duration;
• a 3D convolution unit network to capture a second set of spatio-temporal features by processing the captured optical flow information of the plurality of video frames, wherein the captured second set of spatio-temporal is of the first-time duration;
• concatenate the captured first set of spatio-temporal and the captured second set of spatio-temporal features to get a combined spatio-temporal information of the predefined sequence of video frames;
• a Long Short-Term Memory (LSTM) unit network to process the combined spatio-temporal information to capture a spatio-temporal representation spanning a second-time duration, wherein the second-time duration is longer than the first-time duration;
• reconstruct one or more details from each of the plurality of video frames by feeding the captured spatio-temporal representation and the captured spatial information to a decoder network, wherein the decoder network:
• process the captured spatio-temporal representation using a combination of a 3D convolution unit layers and a 3D transpose convolution unit layers of the decoder network to map the captured spatio-temporal representation to each frame of the video;
• and
• concatenate the captured spatial temporal information from 2D convolution unit network to one or more predefined layers of the decoder network.