Electronic device for at least one of video moment retrieval and highlight detection and operation method thereof

An electronic device for at least one of video moment retrieval and highlight detection, the device comprising:

• a storage unit;
• and
• a processor,
• wherein the processor
• obtains a plurality of first video features each corresponding to a plurality of time points or time periods different from one another from a plurality of video clips of a video by using a video encoder through convolution operations or attention mechanisms,
• obtains a text query feature from a text query by using a text encoder through attention mechanism,
• wherein the processor, by using a cross-attention model,
• obtains a plurality of video queries from the plurality of first video features by performing multi-layer perceptron transformations,
• obtains a text key from the text query feature by performing multi-layer perceptron transformation,
• obtains a text value from the text query feature by performing multi-layer perceptron transformation,
• obtains a plurality of attention scores based upon the plurality of video queries, the text key, and the text value by performing dot product operations between the plurality of video queries and the text key, by performing softmax operations on results of the dot product operations and by performing multiplications between results of the softmax operations and the text value,
• obtains a plurality of weights from the plurality of attention scores by performing multi-layer perceptron transformations,
• obtains a plurality of second video features by multiplying each of the plurality of weights and each of the plurality of first video features,
• wherein the cross-attention model models a relationship between the text query and each of the video clips by correcting the plurality of first video features using the text query feature,
• wherein the processor, by using a transformer encoder, obtains a plurality of third video features and a second saliency token from the plurality of second video features and a first saliency token,
• wherein the processor obtains a plurality of saliency scores for the plurality of video clips based upon the plurality of third video features and the second saliency token,
• wherein the processor modifies the cross-attention model, the transformer encoder and the first saliency token such that at least one saliency score corresponding to a positive pair of the video and the text query is increased and at least one saliency score corresponding to a negative pair of the video and the text query is decreased, in order to reflect relevance of the text query to the video.