Systems and methods to predict aggression in surveillance camera video

A system for predicting aggressive behavior associated with a surveillance scene, the system comprising:

• an image sensor configured to generate images of a surveillance scene;
• an audio sensor configured to generate audio of the surveillance scene;
• a local computing system local to the image sensor and audio sensor, the local computing system configured to:
• execute an object classification model on the images to predict one or more classes of objects in the surveillance scene,
• execute a sound-event detection model on the audio to predict one or more classes of events occurring in the surveillance scene, and
• generate metadata associated with the predicted one or more classes of objects and the predicted one or more classes of events, wherein the generated metadata includes a plurality of predicted classes, each of the plurality of predicted classes having an associated prediction score, and based on the prediction score the object classification model is configured to predict whether an identified person is acting in an out-of-the ordinary manner or anomalous fashion;
• a remote computing system remote from the local computing system, the remote computing system configured to:
• receive the metadata from the local computing system,
• execute a knowledge graph on the metadata, wherein knowledge graph-based reasoning is implemented to predict aggressive behavior occurring in the surveillance scene based on the metadata,
• label the metadata associated with the predicted aggressive behavior as aggressive behavior occurring in the surveillance scene, wherein the labeling yields labeled metadata, and
• output a control command based on the labeled metadata.

A system comprising:

• an image sensor configured to generate images of a surveillance scene;
• an audio sensor configured to generate audio of the surveillance scene;
• and
• a plurality of processors coupled to a plurality of non-transitory computer-readable storage mediums that collectively store programming instructions for execution by the plurality of processors, wherein the programming instructions cause the plurality of processors to:
• execute an object classification model on the images to predict one or more classes of objects in the surveillance scene,
• execute a sound-event detection model on the audio to predict one or more classes of events occurring in the surveillance scene,
• generate metadata associated with the predicted one or more classes of objects and the predicted one or more classes of events, wherein the generated metadata includes a plurality of predicted classes, each of the plurality of predicted classes having an associated prediction score, and based on the prediction score the object classification model is configured to predict whether an identified person is acting in an out-of-the ordinary manner or anomalous fashion,
• execute a knowledge graph on the metadata, wherein knowledge graph-based reasoning is implemented to predict aggressive behavior occurring in the surveillance scene based on the metadata,
• label the metadata associated with the predicted aggressive behavior as aggressive behavior occurring in the surveillance scene, wherein the labeling yields labeled metadata, and
• output a control command based on the labeled metadata.