Incremental video highlights detection system and method

A method for performing incremental highlights detection in video frames, the method comprising steps to:

• receive an input video;
• sample video frames from the input video;
• extract frame-wise spatial features from the video frames using a convolutional neural network;
• extract a frame-wise temporal feature for each video frame;
• aggregate the frame-wise spatial features and the frame-wise temporal feature for each video frame to provide a temporal context to the frame-wise spatial features;
• input the aggregated frame-wise spatial features and the frame-wise temporal features into a transformer encoder to aggregate a temporal context to each frame-wise feature to obtain temporal-aware feature representations of the video frames;
• input the feature representations into a feedforward network model to obtain feedforward-transformed features;
• obtain a parameter by inputting each feedforward-transformed feature and a set of highlight prototypes into a function comparing the feedforward-transformed features to the set of highlight prototypes;
• classify the video frames as highlights or non-highlights based on the parameter calculated using the function comparing the feedforward-transformed features to the set of highlight prototypes;
• and
• output the classification of the video frames, wherein
• the convolutional neural network, the transformer encoder, and the feedforward network model are incrementally trained so that the highlight prototypes are trained from weights learned in previous training stages.

An incremental video highlights detection system comprising:

• a processor and memory of a computing device, the processor being configured to execute a program using portions of memory to:
• receive an input video of video frames;
• extract frame-wise spatial features from the video frames using a convolutional neural network;
• extract a frame-wise temporal feature for each video frame;
• aggregate the frame-wise spatial features and the frame-wise temporal feature for each video frame to provide a temporal context to the frame-wise spatial features;
• input the aggregated frame-wise spatial features and the frame-wise temporal features into a transformer encoder to aggregate a temporal context to each frame-wise feature to obtain temporal-aware feature representations of the video frames;
• input the feature representations into a feedforward network model to obtain feedforward-transformed features;
• calculate a distance between each feedforward-transformed feature and a set of highlight prototypes to obtain highlight pair-wise distances between the feedforward-transformed features and the set of highlight prototypes;
• classify the video frames as highlights based on the obtained highlight pair-wise distances;
• and
• output the classification of the video frames.