Method for recognizing activities using separate spatial and temporal attention weights

A device for recognizing person activity in a video comprising a sequence of frames, each frame showing at least a portion of the person, the device comprising:

• a module for obtaining a set of consecutive 3D poses of the person using the sequence of frames, each of the consecutive 3D poses illustrating a posture of the person from a frame of the sequence of frames, and each of the consecutive 3D poses being associated with an instant in the sequence of frames,
• a first encoder neural network configured to receive the sequence of frames as input and to output the feature map having dimensions associated with time, space, and a number of channels,
• a second neural network configured to receive the set of consecutive 3D poses of the person as input and to output a vector of spatiotemporal features,
• a third neural network configured to receive the vector of spatiotemporal features as input and to output a matrix of spatial attention weights, wherein each weight indicates an importance of a location in the matrix, wherein the third neural network comprises a first fully connected layer, a hyperbolic tangent layer, a second fully connected layer, and a sigmoid layer,
• a fourth neural network, different from the third neural network, configured to receive the vector of spatiotemporal features as input and to output a matrix of temporal attention weights, wherein each weight indicates a saliency of an instant in the sequence of frames, wherein the fourth neural network comprises a first fully connected layer, a hyperbolic tangent layer, a second fully connected layer, and a Softmax layer,
• a module for obtaining a spatially-modulated feature map by modulating the feature map using the matrix of spatial attention weights,
• a module for obtaining a temporally-modulated feature map, different from the spatially-modulated feature map, by modulating the feature map using the matrix of temporal attention weights,
• a module for performing a convolution of the spatially modulated feature map and of the temporally modulated feature map to obtain a convoluted feature map,
• a module for performing a classification using the convoluted feature map so as to determine the activity of the person in the video.