Systems and methods for counting repetitive activity in audio video content
AssigneeInception Institute of Artificial Intelligence, Ltd.
InventorsYunhua Zhang, Cees G. M. Snoek, Ling Shao
Life of this patent
Estimating the term…
Abstract
Repetitive activities can be captured in audio video content. The AV content can be processed in order to predict the number of repetitive activities present in the AV content. The accuracy of the predicted number may be improved, especially for AV content with challenging conditions, by basing the predictions on both the audio and video portions of the AV content.
Claims21
A method for predicting a number of repetitions in an audio/video (AV) content comprising:
- extracting video features from a video portion of the AV content;
- extracting audio features from an audio portion of the AV content;
- processing at least a portion of the extracted video features and a portion of the extracted audio features to determine a temporal stride for use in processing the video portion of the AV content;
- processing the extracted video features with a neural network to predict a number of repetitive actions in the video portion based on the determined temporal stride;
- processing the extracted audio features with a neural network to predict a number of repetitive actions in the audio portion
- predicting a final number of repetitive actions from the predictions of the number of repetitive actions in the video portion and audio portion and the extracted video and audio features.
A non-transitory computer readable memory storing instructions, which when executed by a processor of a system configure the system to perform a method comprising:
- extracting video features from a video portion of the AV content;
- extracting audio features from an audio portion of the AV content;
- processing at least a portion of the extracted video features and a portion of the extracted audio features to determine a temporal stride for use in processing the video portion of the AV content;
- processing the extracted video features with a neural network to predict a number of repetitive actions in the video portion based on the determined temporal stride;
- processing the extracted audio features with a neural network to predict a number of repetitive actions in the audio portion
- predicting a final number of repetitive actions from the predictions of the number of repetitive actions in the video portion and audio portion and the extracted video and audio features.
A system comprising:
- a processor for executing instructions;
- and
- a memory storing instructions, which when executed by the processor configure the system to perform a method comprising:
- extracting video features from a video portion of the AV content;
- extracting audio features from an audio portion of the AV content;
- processing at least a portion of the extracted video features and a portion of the extracted audio features to determine a temporal stride for use in processing the video portion of the AV content;
- processing the extracted video features with a neural network to predict a number of repetitive actions in the video portion based on the determined temporal stride;
- processing the extracted audio features with a neural network to predict a number of repetitive actions in the audio portion
- predicting a final number of repetitive actions from the predictions of the number of repetitive actions in the video portion and audio portion and the extracted video and audio features.
Dependencies are inferred from the claim text.