Edge-enabled system for smart video processing, vectorized object detection, search, and customizable alerts using smart nvr devices

A computer implemented method for video surveillance, comprising:

• capturing, by a plurality of cameras, video data;
• for each of one or more network video recorders (NVRs) connected to the plurality of cameras:
• processing the video data to generate, for segments of the video data, machine readable representations comprising attribute data describing visual attributes of detected objects or events and textual descriptions of the segments;
• converting the machine readable representations into vectorized embeddings representing semantic attributes of the detected objects or events;
• and
• transmitting the vectorized embeddings and associated metadata to a remote vector database (Vector DB) hosted on a cloud based server;
• storing, by the Vector DB, the vectorized embeddings and indexing the vectorized embeddings for similarity based searches;
• receiving, by at least one processor of the cloud based server, a user query in natural language;
• processing, by the at least one processor, the user query with a language model to produce one or more vectorized search embeddings;
• executing, by the at least one processor over the Vector DB, a multi-level iterative similarity search using the one or more vectorized search embeddings, wherein each level of the multi-level iterative similarity search filters candidate vector embeddings identified by a previous level;
• and
• identifying, by the at least one processor, video segments that are semantically relevant to the user query based on results of the multi-level iterative similarity search and causing presentation of the identified video segments together with corresponding temporal and spatial metadata.

A non-transitory computer readable medium storing instructions that, when executed by one or more processors of a video surveillance system comprising a plurality of cameras, one or more network video recorders (NVRs), and a cloud-based server hosting a remote vector database (Vector DB), cause the one or more processors to perform operations comprising:

• receiving, at the one or more NVRs, captured video data from the plurality of cameras;
• for each of the one or more NVRs:
• processing the video data to generate, for segments of the video data, machine readable representations comprising attribute data describing visual attributes of detected objects or events and textual descriptions of the segments;
• converting the machine readable representations into vectorized embeddings representing semantic attributes of the detected objects or events;
• and
• transmitting the vectorized embeddings and associated metadata to the remote Vector DB hosted on the cloud-based server;
• storing, by the Vector DB, the vectorized embeddings and indexing the vectorized embeddings for similarity based searches;
• receiving, by at least one processor of the cloud-based server, a user query in natural language;
• processing, by the at least one processor of the cloud-based server, the user query with a language model to produce one or more vectorized search embeddings;
• executing, by the at least one processor of the cloud-based server over the Vector DB, a multi-level iterative similarity search using the one or more vectorized search embeddings, wherein each level of the multi-level iterative similarity search filters candidate vector embeddings identified by a previous level;
• and
• identifying, by the at least one processor of the cloud-based server, video segments that are semantically relevant to the user query based on results of the multi-level iterative similarity search and causing presentation of the identified video segments together with corresponding temporal and spatial metadata.