3D convolutional neural networks for television advertisement detection

A computer implemented method for classifying whether video content is likely to be an advertisement, the video content including a plurality of image frames the method comprising:

• (a) providing a curated database of video content items that includes (i) a plurality of different video content items that were each previously identified as being an advertisement, and (ii) a plurality of different video content items that were each previously identified as not being an advertisement;
• (b) training a two-dimensional convolutional neural network (2D CNN) using the curated database of video content items, wherein the training includes learning characteristic visual and spatial features of advertisement images in the video content items compared to non-advertisement images in the video content items, the training resulting in weights being defined for the 2D CNN;
• (c) training a three-dimensional convolutional neural network (3D CNN) using the curated database of video content items, wherein the training includes learning a temporal structure and relationship over multiple image frames of the advertisements in the video content items compared to non-advertisement image frames in the video content items, the training resulting in weights being defined for the 3D CNN;
• (d) sampling N image frames in video content that is desired to be classified;
• (e) using the trained 2D CNN to extract visual and spatial features from the N image frames, and passing the extracted visual and spatial features from the N image frames through a first fully connected layer to extract a first representation of the N image frames;
• (f) using the trained 3D CNN to capture temporal structure and relationship information from the N image, frames, and passing the captured temporal structure and relationship information through a second fully connected layer to extract a second representation of the N image frames;
• (g) aggregating the first and second representations of the N image frame to obtain a final representation of the N image frames;
• and
• (h) passing the final representation of the N image frames through a final fully connected layer and then through a classifier, the classifier outputting a probability that the video content is an advertisement.