US 2026/0127166 A1Published application

VideoRAG using Natural Language as Intermediate Representation in Multi-Camera, Closed-Domain Applications

AssigneeShobdo, LLC

InventorsFaria Azim, Ikra Iftekhar Shuvo

Life of this patent

Estimating the term…

Loading patent…

Abstract

A Video Retrieval Augmented Generation (VideoRAG) system for closed-domain applications that uses natural language text as an intermediate representation between video content and query systems. A unified vision-language model (VLM) processes video frames and generates structured JSON text descriptions conforming to domain-specific event schemas, while simultaneously answering natural language queries through retrieval-augmented generation. The natural language intermediate representation provides substantial storage efficiency improvements over embedding-based approaches, human-interpretable analytics capabilities, and cross-camera entity tracking. The architecture supports a closed-domain applications including but not limited to retail analytics, healthcare monitoring, and industrial safety operations.

Claims23

Dependent claims

A computer-implemented method for video retrieval augmented generation using natural language intermediate representation, comprising:

(a) Scene Sensing Module—Processing sampled video frames to:
(i) detect meaningful motion events while filtering environmental motion comprising at least curtain movements and shadow movements,
(ii) generate sensory logs comprising crude entity descriptions in structured natural language format, wherein said crude entity descriptions follow a pattern entity_type, and
(iii) route disaster alerts comprising at least fire, smoke, or flood detections to an alert pathway;
(b) Entity Trail Logger—Employing a reasoning-capable vision-language model to:
(i) analyze said sensory logs using thought tokens comprising internal reasoning mechanisms to select relevant video clips for detailed examination,
(ii) generate entity trail logs comprising clip-based temporal segments, wherein each entity trail log entry specifies:
an entity identifier, a camera identifier, a clip start time, a clip end time, and a natural language description of entity activities within said temporal segment,
(iii) identify key frames within each clip-based temporal segment where entities are most identifiable,
(iv) track entities across multiple camera views by computing weighted attribute correlation scores between entities appearing in different cameras, wherein said weighted correlation comprises clothing attribute similarity, height similarity, and temporal proximity,
(v) store both sensory logs and entity trail logs in a dual-schema storage architecture, wherein sensory logs are stored in chronologically-sorted files for sequential access and entity trail logs are stored in a database indexed using a temporal B-tree index and an entity-reference inverted index generated from said key frames, and
(vi) delay indexing of entity trail log entries until entity state remains unchanged for a predetermined stabilization duration, preventing premature indexing of incomplete entity trails;
(c) User-facing Agent—Processing natural language queries to:
(i) employ thought tokens comprising internal reasoning mechanisms to analyze query intent and determine which log types to search,
(ii) retrieve relevant log entries from said sensory logs and said entity trail logs using hybrid sparse-dense text matching algorithms comprising BM25 for sparse lexical matching and sentence embeddings for dense semantic matching,
(iii) determine relevant video segments based on retrieved log entries, and
(iv) generate natural language responses with mandatory citations to source events and video timestamps, wherein said responses are derived exclusively from said retrieved log entries.

14 more elements

A video retrieval augmented generation apparatus implementing natural language intermediate representation architecture, comprising:

(a) Scene sensing processor configured to:
(i) process sampled video frames to detect meaningful motion events while filtering environmental motion,
(ii) generate sensory logs comprising crude entity descriptions in structured natural language format, and
(iii) route disaster alerts to an alert pathway;
(b) Entity trail logger processor with reasoning capabilities, configured to:
(i) analyze said sensory logs using thought tokens to select relevant video clips,
(ii) generate entity trail logs comprising clip-based temporal segments with entity identifiers, camera identifiers, clip start times, clip end times, and natural language activity descriptions,
(iii) identify key frames where entities are most identifiable,
(iv) track entities across cameras using weighted attribute correlation,
(v) implement a dual-schema storage architecture storing sensory logs in chronologically-sorted files for sequential access and entity trail logs in a database with temporal B-tree indexing and entity-reference inverted index generated from key frames, and
(vi) implement a delayed indexing mechanism that indexes entity trail log entries only after entity state remains unchanged for a predetermined stabilization duration;
(c) User-facing agent processor with reasoning capabilities, configured to:
(i) employ thought tokens to analyze query intent and determine log types to search,
(ii) implement a hybrid retrieval subsystem performing sparse lexical matching using BM25 and dense semantic matching using sentence embeddings on said sensory logs and said entity trail logs,
(iii) determine relevant video segments based on retrieved logs, and
(iv) generate natural language responses with mandatory citations to source events and video timestamps.

13 more elements

Dependencies are inferred from the claim text.