System and method for detecting video semantic interval

A system for detecting a video semantic interval, comprising:

• a communication module configured to receive a video and a query sentence;
• memory in which a program for outputting a semantic interval proposal from the video and the query sentence is stored;
• and
• a processor configured to execute the program stored in the memory,
• wherein by executing the program, the processor outputs a semantic interval proposal having start timing and end timing, which is matched with the query sentence within the video over a pre-trained video semantic interval detection network based on boundary refinements as results of a detection of the semantic interval proposal, and outputs a semantic interval proposal having a variable boundary through refinements of a predetermined semantic interval proposal, and
• wherein the processor operates a proposal feature extraction unit comprising:
• a unimodal feature encoding unit configured to extract unimodal features from the received video and query sentence, respectively;
• a multimodal feature encoding unit configured to extract a multimodal feature in which the features of the received video and query sentence interact with each other;
• a cross-modal proposal feature map extraction unit configured to extract a feature map of the semantic interval proposal based on the unimodal features and the multimodal feature;
• a proposal interval length similarity map extraction unit configured to extract a proposal interval length similarity map based on the multimodal feature;
• and
• a proposal interaction feature map extraction unit configured to concatenate the feature map of the semantic interval proposal and the proposal interval length similarity map and to extract a proposal interaction feature map through an interaction between a plurality of the semantic interval proposals.