System and method for group activity recognition in images and videos with self-attention mechanisms
AssigneeSportlogiq Inc.
InventorsMehrsan Javan Roshtkhari, Kirill Gavrilyuk, Ryan Hartley Sanford
Life of this patent
Estimating the term…
Abstract
A system and method are described, for automatically analyzing and understanding individual and group activities and interactions. The method includes receiving at least one image from a video of a scene showing one or more individual objects or humans at a given time; applying at least one machine learning or artificial intelligence technique to automatically learn a spatial, temporal or a spatio-temporal informative representation of the image and video content for activity recognition; and identifying and analyzing individual and group activities in the scene.
Claims20
A method for processing visual data for individual and group activities and interactions, the method comprising:
- receiving a single raw video of a scene showing one or more entities over a given time;
- obtaining a training set comprising at least one labeled individual or group activity;
- training at least one machine learning or artificial intelligence technique using the obtained at least one labeled individual or group activity in the training set to directly process the received single raw video by automatically representing spatial, temporal or spatio-temporal content of the visual data of the received single raw video and numerically modelling temporally static and temporally dynamic features of each of the one or more entities in the single raw video by generating numerical representations capturing the temporally static and temporally dynamic features;
- inferring at least one individual or group activity based on the generated numerical representations;
- and
- localizing the at least one individual or group activity in space and time in a plurality of images of the single raw video.
A non-transitory computer readable medium storing computer executable instructions for processing visual data for individual and group activities and interactions, comprising instructions for:
- receiving a single raw video of a scene showing one or more entities over a given time;
- obtaining a training set comprising at least one labeled individual or group activity;
- training at least one machine learning or artificial intelligence technique using the obtained at least one labeled individual or group activity in the training set to directly process the received single raw video by automatically representing spatial, temporal or spatio-temporal content of the visual data of the received single raw video and numerically modelling temporally static and temporally dynamic features of each of the one or more entities in the single raw video by generating numerical representations capturing the temporally static and temporally dynamic features;
- inferring at least one individual or group activity based on the generated numerical representations;
- and
- localizing the at least one individual or group activity in space and time in a plurality of images of the single raw video.
A device configured to process visual data for individual and group activities and interactions, the device comprising a processor and memory, the memory storing computer executable instructions that, when executed by the processor, cause the device to:
- receive a single raw video of a scene showing one or more entities over a given time;
- obtain a training set comprising at least one labeled individual or group activity;
- train at least one machine learning or artificial intelligence technique using the obtained at least one labeled individual or group activity in the training set to directly process the received single raw video by automatically representing spatial, temporal or spatio-temporal content of the visual data of the received single raw video and numerically modelling temporally static and temporally dynamic features of each of the one or more entities in the single raw video by generating numerical representations capturing the temporally static and temporally dynamic features;
- infer at least one individual or group activity based on the generated numerical representations;
- and
- localize the at least one individual or group activity in space and time in a plurality of images of the single raw video.
Dependencies are inferred from the claim text.