Surveillance system using adaptive spatiotemporal convolution feature representation with dynamic abstraction for video to language translation

A computer-implemented method for surveillance, comprising:

• capturing, by an image capture device, a video sequence of a target area that includes one or more objects and is formed from a set of image frames;
• applying, by a processor, a three-dimensional Convolutional Neural Network (C3D) to the image frames to obtain, for the video sequence, (i) intermediate feature representations across L convolutional layers and (ii) top-layer features,
• producing, by the processor, a first word of an output caption for the video sequence by applying the top-layer features to a Long Short Term Memory (LSTM),
• producing, by the processor, subsequent words of the output caption by (i) dynamically performing spatiotemporal attention and layer attention using the intermediate feature representations to form a context vector, and (ii) applying the LSTM to the context vector, a previous word of the output caption, and a hidden state of the LSTM;
• and
• displaying, by a display device, the output caption to a user, the output caption describing an activity of the one or more objects in the target area.

A computer program product for surveillance, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:

• capturing, by an image capture device operatively coupled to the computer, a video sequence of a target area that includes one or more objects and is formed from a set of image frames;
• applying, by a processor of the computer, a three-dimensional Convolutional Neural Network (C3D) to the image frames to obtain, for the video sequence, (i) intermediate feature representations across L convolutional layers and (ii) top-layer features,
• producing, by the processor, a first word of an output caption for the video sequence by applying the top-layer features to a Long Short Term Memory (LSTM),
• producing, by the processor, subsequent words of the output caption by (i) dynamically performing spatiotemporal attention and layer attention using the intermediate feature representations to form a context vector, and (ii) applying the LSTM to the context vector, a previous word of the output caption, and a hidden state of the LSTM;
• and
• displaying, by a display device operatively coupled to the computer, the output caption to a user, the output caption describing an activity of the one or more objects in the target area.