System and Method for Detecting and Explaining Anomalies in Video of a Scene

A system for video anomaly detection, comprising:

• a processor;
• and a memory having instructions stored thereon that, when executed by the processor, cause the system to:
• collect a sequence of input video frames of an input video of a scene;
• partition the sequence of input video frames into a plurality of input video patches, wherein each of the plurality of input video patches is a spatio-temporal patch;
• process each of the plurality of input video patches with one or more classifiers, wherein each of the one or more classifiers corresponds to a deep neural network having an output layer trained to estimate one or more attributes of the plurality of input video patches from an output of a penultimate layer of the deep neural network;
• compare the output of the penultimate layer of the one or more classifiers generated using the plurality of input video patches with nominal outputs of the penultimate layer of the one or more classifiers generated using a plurality of nominal video patches from corresponding spatial regions,
• wherein the plurality of nominal video patches are extracted from nominal video of the scene
• detect an anomaly when the output of the penultimate layers of the one or more classifiers for an input video patch is dissimilar to the outputs of the penultimate layers of the one or more classifiers for the plurality of nominal video patches from the same spatial region as the input video patch;
• and
• providing an output comprising an explanation of a type of the detected anomaly, wherein the output is provided based on the one or more attributes of the input video patch estimated by the output layer of the one or more classifiers that are dissimilar to the attributes of the closest matching nominal video patch.