# US 12,182,198 B2 · Systems and methods for identifying events within video content using intelligent search query > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,198 B2 - Document ID: US12182198B2 - Status: Granted - Kind: B2 - Application number: 18/112,926 - Filed: 2023-02-22 - Published: 2023-06-29 - Granted: 2024-12-31 - Earliest claimed: 2020-02-17 - Assignees: Honeywell International Inc. - Applicants: Honeywell International Inc. - Inventors: Kirupakar Janakiraman; Baskaran Muthusamy; Keyurbhai Patel - Web page: https://patentgrep.com/patent/US12182198B2 ## CPC - G06F 16/7837: {using objects detected or recognised in the video content} - G06F 16/732: Query formulation - G06F 16/735: Filtering based on additional data, e.g. user or group profiles - G06F 16/738: Presentation of query results - G06F 16/7867: {using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings} - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06N 20/00: Machine learning - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames} - G06V 20/48: {Matching video sequences} - H04L 65/612: for unicast - H04L 65/765: {intermediate} - G06V 20/44: {Event detection} - H04L 65/61: for supporting one-way streaming services, e.g. Internet radio - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract A video management system (VMS) may search for one or more objects and/or events in one or more video streams, and may receive time-stamped metadata that may identify one or more objects and/or events occurring in the corresponding video stream as well as an identifier that uniquely identifies the corresponding video stream. A user may enter a query into a video query engine, wherein the video query engine includes one or more cognitive models. The VMS may apply the search query to the time-stamped metadata via the video query engine to search for one or more objects and/or events in the one or more video streams that match the search query, and returning a search result to the user. ## Claims ### Claim 1 A system for searching for one or more objects and/or events in one or more video streams, the system comprising: a site including one or more video cameras, each of the one or more video cameras capturing a corresponding video stream, the site including one or more memories for storing each of the one or more video streams and time-stamped metadata for each of the one or more video streams, the time-stamped metadata for each video stream identifying one or more objects and/or events occurring in the corresponding video stream as well as an identifier that uniquely identifies the corresponding video stream, the one or more memories further storing time-stamped data provided by one or more non-video based sources; a central hub operatively coupled to the site, the central hub configured to receive at least some of the time time-stamped metadata from the site and at least some of the time-stamped data provided by the one or more non-video based sources; the central hub including a video query engine that is configured to: receive a user query from a user via a user interface, wherein the user query is in a natural human language format; process the user query in combination with at least some of the time-stamped data provided by the one or more non-video based sources to identify an inference for the user query, wherein the inference infers a situational context under which the user query was entered; build a search query in a computer search language, wherein the search query is based at least in part on both the user query and the situational context inferred by the identified inference; apply the search query to the time-stamped metadata for each of the one or more video streams stored in the memory to search for one or more objects and/or events in the one or more video streams that match the search query; return a search result to the user, wherein the search result identifies one or more matching objects and/or events in the one or more video streams that match the search query, and for each matching object and/or event that matches the search query, providing a reference to the corresponding video stream stored at the site and a reference time in the corresponding video stream that includes the matching object and/or event; the central hub downloading a video clip from the site that includes the matching object and/or event; and the user interface displaying the downloaded video clip that includes the matching object and/or event. ### Claim 2 (depends on 1) The system of claim 1, wherein non-video based sources comprise one or more sensors. ### Claim 3 (depends on 2) The system of claim 2, wherein the one or more sensors comprises one or more security sensors. ### Claim 4 (depends on 1) The system of claim 1, wherein the inference infers a user's intent of the user query based at least in part on the user query and the situational context under which the user query was entered, and wherein the video query engine is configured to build the search query based at least in part on the user query and the user's intent inferred by the identified inference. ### Claim 5 (depends on 1) The system of claim 1, wherein the inference infers a user's emotional state based at least in part on the user query and the situational context under which the user query was entered, and wherein the video query engine is configured to build the search query based at least in part on the user query and the user's emotional state inferred by the identified inference. ### Claim 6 (depends on 1) The system of claim 1, wherein the video query engine is configured to process the user query, at least some of the time-stamped data provided by one or more non-video based sources and at least some of the time-stamped metadata for one or more of the one or more video streams to identify the inference for the user query, wherein the inference infers a situational context under which the user query was entered. ### Claim 7 (depends on 1) The system of claim 1, wherein the video query engine uses machine learning to infer the situational context under which the user query was entered. ### Claim 8 (depends on 7) The system of claim 7, wherein the machine learning of the video query engine is refined over time based on user feedback. ### Claim 9 (depends on 8) The system of claim 8, wherein the user feedback includes a subsequent modified user query that is entered by the user after the video query engine returns the search result to the user. ### Claim 10 A system for searching for one or more objects and/or events in one or more video streams, the system comprising: a site including one or more video cameras, each of the one or more video cameras capturing a corresponding video stream, the site including one or more memories for storing each of the one or more video streams and time-stamped metadata for each of the one or more video streams, the time-stamped metadata for each video stream identifying one or more objects and/or events occurring in the corresponding video stream as well as an identifier that uniquely identifies the corresponding video stream; a central hub operatively coupled to the site, the central hub configured to receive at least some of the time time-stamped metadata from the site and at least some time-stamped data provided by the one or more non-video based sources; the central hub including a video query engine that is configured to: receive a user query from a user via a user interface, wherein the user query is in a natural human language format; process the user query in combination with at least some of the time-stamped metadata for one or more of the one or more video streams and at least some time-stamped data provided by the one or more non-video based sources to identify an inference for the user query, wherein the inference infers a situational context under which the user query was entered; build a search query in a computer search language, wherein the search query is based at least in part on both the user query and the situational context inferred by the identified inference; apply the search query to the time-stamped metadata for each of the one or more video streams stored in the memory to search for one or more objects and/or events in the one or more video streams that match the search query; return a search result to the user, wherein the search result identifies one or more matching objects and/or events in the one or more video streams that match the search query, and for each matching object and/or event that matches the search query, providing a reference to the corresponding video stream stored at the site and a reference time in the corresponding video stream that includes the matching object and/or event; and the central hub downloading a video clip from the site that includes the matching object and/or event; and the user interface displaying the downloaded video clip that includes the matching object and/or event. ### Claim 11 (depends on 10) The system of claim 10, wherein the inference infers a user's intent of the user query based at least in part on the user query and the situational context under which the user query was entered, and wherein the video query engine is configured to build the search query based at least in part on the user query and the user's intent inferred by the identified inference. ### Claim 12 (depends on 10) The system of claim 10, wherein the inference infers a user's emotional state based at least in part on the user query and the situational context under which the user query was entered, and wherein the video query engine is configured to build the search query based at least in part on the user query and the user's emotional state inferred by the identified inference. ### Claim 13 (depends on 10) The system of claim 10, wherein the video query engine uses machine learning to infer the situational context under which the user query was entered. ### Claim 14 (depends on 13) The system of claim 13, wherein the machine learning of the video query engine is refined over time based on user feedback. ### Claim 15 (depends on 14) The system of claim 14, wherein the user feedback includes a subsequent modified user query that is entered by the user after the video query engine returns the search result to the user. ### Claim 16 A method for searching for one or more objects and/or events in one or more video streams, the method comprising: capturing, at a remote site, one or more video streams, and storing each of the one or more video streams at the remote site; generating and storing at the remote site time-stamped metadata for each of the one or more video streams, the time-stamped metadata for each video stream identifying one or more objects and/or events occurring in the corresponding video stream as well as an identifier that uniquely identifies the corresponding video stream; a central hub receiving from the remote site at least some of the time-stamped metadata for one or more of the video streams captured at the remote site; receiving, at the central hub, time-stamped data provided by one or more non-video based sources; receiving a user query, wherein the user query is in a natural human language format; the central hub providing the user query to a video query engine, the video query engine: processing the user query in combination with at least some of the time-stamped metadata provided by one or more non-video based sources and at least some of the time-stamped metadata for one or more of the one or more video streams to identify an inference for the user query, wherein the inference infers a situational context under which the user query was entered; building a search query in a computer search language, wherein the search query is based at least in part on both the user query and the situational context inferred by the identified inference; applying the search query to the time-stamped metadata for one or more of the video streams to search for one or more objects and/or events in the one or more video streams that matches the search query; returning a search result to the user, wherein the search result identifies one or more matching objects and/or events in the one or more video streams that match the search query; the central hub downloading from the site a video clip that includes one or more of the matching objects and/or events; and displaying the downloaded video clip on a display. ### Claim 17 (depends on 16) The method of claim 16, further comprising processing the time-stamped metadata for the one or more video streams to identify contextual relationships between objects and/or events occurring in the one or more video streams before entering the user query. ### Claim 18 (depends on 16) The method of claim 16, further comprising inferring a user's intent of the user query based at least in part on the user query and the situational context under which the user query was entered; and building the search query based at least in part on the user query and the user's intent. ### Claim 19 (depends on 16) The method of claim 16, further comprising inferring a user's emotional state based at least in part on the user query and the situational context under which the user query was entered; and building the search query based at least in part on the user query and the user's emotional state.