Video retrieval system using adaptive spatiotemporal convolution feature representation with dynamic abstraction for video to language translation

A computer-implemented method for video retrieval, comprising:

• retrieving, by a set of servers, a video sequence from a database of multiple video sequences and forwarding the video sequence to a requesting hardware device responsive to a match between an input text provided by a user of the requesting hardware device and a video caption for the video sequence,
• wherein the method further comprises translating, by the set of servers, the video sequence into the video caption by
• applying a three-dimensional Convolutional Neural Network (C3D) to image frames of the video sequence to obtain, for the video sequence, (i) intermediate feature representations across L convolutional layers and (ii) top-layer features,
• producing a first word of the video caption for the video sequence by applying the top-layer features to a Long Short Term Memory (LSTM), and
• producing subsequent words of the video caption by (i) dynamically performing spatiotemporal attention and layer attention using the intermediate feature representations to form a context vector, and (ii) applying the LSTM to the context vector, a previous word of the video caption, and a hidden state of the LSTM.

A computer program product for video retrieval, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:

• retrieving, by a set of servers, a video sequence from a database of multiple video sequences and forwarding the video sequence to a requesting hardware device responsive to a match between an input text provided by a user of the requesting hardware device and a video caption for the video sequence,
• wherein the method further comprises translating, by the set of servers, the video sequence into the video caption by
• applying a three-dimensional Convolutional Neural Network (C3D) to image frames of the video sequence to obtain, for the video sequence, (i) intermediate feature representations across L convolutional layers and (ii) top-layer features,
• producing a first word of the video caption for the video sequence by applying the top-layer features to a Long Short Term Memory (LSTM), and
• producing subsequent words of the video caption by (i) dynamically performing spatiotemporal attention and layer attention using the intermediate feature representations to form a context vector, and (ii) applying the LSTM to the context vector, a previous word of the video caption, and a hidden state of the LSTM.