Compositional reasoning of gorup activity in videos with keypoint-only modality

A non-transitory computer-readable storage medium comprising a computer-readable program for compositional reasoning of group activity in videos with keypoint-only modality, wherein the computer-readable program when executed on a computer causes the computer to perform the steps of:

• obtaining video frames from a video stream received from a plurality of video image capturing devices;
• extracting keypoints all of persons detected in the video frames to define keypoint data;
• tokenizing the keypoint data with time and segment information;
• clustering groups of keypoint persons in the video frames and passing the clustering groups through multi-scale prediction;
• and
• performing a prediction to provide a group activity prediction of a scene in the video frames.

A system for compositional reasoning of group activity in videos with keypoint-only modality, the system comprising:

• a memory;
• and
• one or more processors in communication with the memory configured to:
• obtain video frames from a video stream received from a plurality of video image capturing devices;
• extract keypoints all of persons detected in the video frames to define keypoint data;
• tokenize the keypoint data with time and segment information;
• cluster groups of keypoint persons in the video frames and pass the clustering groups through multi-scale prediction;
• and
• perform a prediction to provide a group activity prediction of a scene in the video frames.