Video anomaly detection based on object interactions

A system for video anomaly detection in a scene, comprising:

• an input interface configured to accept an input video of the scene;
• a memory configured to store a set of exemplars, each exemplar in the set of exemplars includes features that are extracted from tracked bounding boxes of a first object tracked in different frames of a normal video of the scene and combined with features of a second object tracked in the different frames of the normal video of the scene, such that the second object is in proximity of the tracked bounding boxes of the first object;
• a processor configured to
• track the first object moving in the input video of the scene by detecting bounding boxes of the first object in a set of frames of the input video;
• extract combined features of the first object in the scene from the detected bounding boxes, the combined features indicative of at least:
• an appearance of the first object in the scene, a size of the first object in the scene, a location of the first object in the scene, and a trajectory of the first object within the scene;
• extract context features of the second object in the scene, from the set of frames of the input video in proximity of the bounding boxes of the first object, the context features of the second object indicative of an appearance of the second object in the scene, a size of the second object in the scene, a location of the second object in the scene, and a trajectory of the second object within the scene;
• generate an input feature vector by combining the combined features of the first object and the context features of the second object;
• determine a smallest distance from the input feature vector to its closest exemplar based on a comparison of the input feature vector with each exemplar in the set of exemplars;
• and
• output an anomaly detection result based on a comparison of the smallest distance with a threshold, wherein the anomaly detection result is indicative of a presence of an anomaly or an absence of the anomaly, for performing the video anomaly detection in the scene.

A method for video anomaly detection in a scene, comprising:

• accepting an input video of the scene;
• tracking an object moving in the input video of the scene by detecting bounding boxes of the object in a set of frames of the input video;
• extracting combined features of the object in the scene from the detected bounding boxes, the combined features indicative of at least:
• an appearance of the object in the scene, a size of the object in the scene, a location of the object in the scene, and a trajectory of the object within the scene;
• extracting context features from the set of frames of the input video in proximity of the detected bounding boxes, the context features indicative of context of the object in the scene;
• generating an input feature vector for the object by combining the combined features with the context features;
• determining a smallest distance from the input feature vector to a corresponding closest exemplar based on a comparison of the input feature vector with each exemplar in a set of exemplars, wherein the each exemplar includes features that are extracted from tracked bounding boxes of an object tracked in different frames of a normal video of the scene and combined with features representing a context associated with the object, the features representing the context are extracted from bounding boxes in proximity of the tracked bounding boxes;
• and
• outputting an anomaly detection result based on a comparison of the smallest distance with a threshold, wherein the anomaly detection result is indicative of a presence of an anomaly or an absence of the anomaly in the scene.