Activity recognition in dark video based on both audio and video content

A non-transitory computer readable memory storing instructions which when executed by a processor of a computer system configure the computer system to perform a method comprising:

• receiving the media content comprising video content and audio content;
• extracting video features from at least a portion of the video content using a video model;
• extracting audio features from at least a portion of the audio content using an audio model;
• generating a darkness-aware feature by applying the video features to a darkness-aware evaluation model, the darkness-aware feature providing an indication of the ability to extract discriminative information from the video content;
• modulating the video features and the audio features using the darkness-aware feature to dynamically adjust an amount of noise filtered out according to illumination of the media content;
• and
• predicting an activity occurring in the media content using the modulated video and audio features by:
• generating an audio prediction by applying the modulated audio features to an audio classifier;
• generating a video prediction by applying the modulated video features to a video classifier;
• and
• combining the audio prediction and the video prediction to predict the activity occurring in the media content,
• wherein the darkness-aware feature is generated according to:
• fd=Φ′d(Fv),
• where:
• fd is the darkness-aware feature;
• Φ′d denotes the darkness-aware evaluation model obtained by removing a fully connected layer from Φd;
• Φd comprises a 3D convolutional network followed by the fully connected layer, Φd trained as a binary classification problem with the fully connected layer outputting a single score D∈ indicating whether visual content can provide discriminative features;
• and
• Fv is the extracted video features.

A computer system comprising:

• a processor for executing instructions;
• and
• a memory storing instructions, which when executed by the processor configure the computer system to perform a method comprising:
• receiving the media content comprising video content and audio content;
• extracting video features from at least a portion of the video content using a video model;
• extracting audio features from at least a portion of the audio content using an audio model;
• generating a darkness-aware feature by applying the video features to a darkness-aware evaluation model, the darkness-aware feature providing an indication of the ability to extract discriminative information from the video content;
• modulating the video features and the audio features using the darkness-aware feature to dynamically adjust an amount of noise filtered out according to illumination of the media content;
• and
• predicting an activity occurring in the media content using the modulated video and audio features by:
• generating an audio prediction by applying the modulated audio features to an audio classifier;
• generating a video prediction by applying the modulated video features to a video classifier;
• and
• combining the audio prediction and the video prediction to predict the activity occurring in the media content,
• wherein the darkness-aware feature is generated according to:
• fd=Φ′d(Fv),
• where:
• fd is the darkness-aware feature;
• Φ′d denotes the darkness-aware evaluation model obtained by removing the fully connected layer from Φd;
• Φd comprises a 3D convolutional network followed by a fully connected layer, Φd trained as a binary classification problem with the fully connected layer outputting a single score D∈ indicating whether visual content can provide discriminative features;
• and
• Fv is the extracted video features.