# US 7,746,378 B2 · Video analysis, archiving and alerting methods and apparatus for a distributed, modular and extensible video surveillance system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 7,746,378 B2 - Document ID: US7746378B2 - Status: Granted - Kind: B2 - Application number: 10/967,048 - Filed: 2004-10-14 - Published: 2006-04-13 - Granted: 2010-06-29 - Earliest claimed: 2004-10-12 - Assignees: International Business Machines Corporation - Inventors: Chiao-Fe Shu; Arun Hampapur; Max Lu - Web page: https://patentgrep.com/patent/US7746378B2 ## CPC - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - G06F 16/738: Presentation of query results - G06F 16/78: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually - G06F 16/7837: {using objects detected or recognised in the video content} - G06F 16/786: {using motion, e.g. object motion or camera motion} - G08B 13/19608: {Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position} - G08B 13/19613: {Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion} - G08B 13/19656: {Network used to communicate with a camera, e.g. WAN, LAN, Internet} - G08B 13/19673: {Addition of time stamp, i.e. time metadata, to video stream} - G08B 13/19682: {Graphic User Interface [GUI] presenting system data to the user, e.g. information on a screen helping a user interacting with an alarm system} - G08B 13/19693: {using multiple video sources viewed on a single or compound screen} - H04N 7/181: {for receiving images from a plurality of remote sources} - H04N 21/2187: Live feed - H04N 21/23109: {by placing content in organized collections, e.g. EPG data repository (details of retrieval of video data and associated meta data in video databases G06F16/739)} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS ## Abstract Systems and methods are disclosed for analyzing, managing and cataloguing video tracks generated by a video surveillance system. Systems and methods operating in accordance with the present invention receive video tracks from a video surveillance system; analyze them for particular categories of, for example, objects or events, assign unique identifying tags to the video tracks; and save them in highly-cross-referenced databases so that video tracks containing, for example, particular objects or events can be easily searched for and recalled. In addition, the present invention provides browsing facilities for viewing events that have just elapsed, events that are occurring in near real-time. ## Claims ### Claim 1 A video surveillance system comprising: at least one video surveillance system module comprising: a video surveillance camera, wherein the video surveillance camera is remotely programmable; a video encoder to encode video gathered by the video surveillance camera; and a video analysis engine coupled to the video surveillance camera to analyze video gathered by the video surveillance camera and to create data derived from the video, where the video analysis engine comprises a track creation system to generate at least one track in dependence on at least one user-specified video analysis criterion, wherein the at least one track comprises at least one pointer to a portion of the video gathered by the video surveillance system satisfying the specified video analysis criterion; at least one video surveillance system network coupled to the at least one video surveillance system module; at least one database and application system coupled to the at least one video surveillance system network to store video gathered by the video surveillance camera and data created by the video analysis engine, and to receive and process queries for the video and data; a video surveillance system control interface coupled to the at least one video surveillance system network to control the operations of the video surveillance system; and a video surveillance application and control interface coupled to the at least one video surveillance system network for controlling video surveillance activities, the video surveillance application and control interface comprising: a program for controlling video surveillance activities; a display for displaying video gathered by the video surveillance camera and data created by the video analysis engine; and at least one input device for creating queries of the database and application system. ### Claim 2 (depends on 1) The video surveillance system of claim 1 wherein: the video analysis engine further comprises: a track identification system for assigning a unique identifier to the at least one track. ### Claim 3 (depends on 2) The video surveillance system of claim 2 wherein the track further comprises track summary information. ### Claim 4 (depends on 3) The video surveillance system of claim 3 where the track identification system applies the unique identifier assigned to the at least one track to the track summary information associated with the at least one track. ### Claim 5 (depends on 4) The video surveillance system of claim 4 where the track summary information further comprises a pointer to a video segment. ### Claim 6 (depends on 4) The video surveillance system of claim 4 where the track summary information further comprises a pointer to a keyframe. ### Claim 7 (depends on 3) The video surveillance system of claim 3 wherein the track summary information further comprises timestamp information memorializing a creation time of the track. ### Claim 8 (depends on 3) The video surveillance system of claim 3 wherein the track summary information further comprises object class information specifying an object class. ### Claim 9 (depends on 8) The video surveillance system of claim 8 where the object class corresponds to a vehicle class. ### Claim 10 (depends on 8) The video surveillance system of claim 8 where the object class corresponds to a human class. ### Claim 11 (depends on 3) The video surveillance system of claim 3 where the track summary information further comprises object area information. ### Claim 12 (depends on 3) The video surveillance system of claim 3 where the track summary information further comprises object velocity information. ### Claim 13 (depends on 3) The video surveillance system of claim 3 where the track summary information further comprises activity class information. ### Claim 14 (depends on 13) The video surveillance system of claim 13 where the activity class information corresponds to loitering activity. ### Claim 15 (depends on 3) The video surveillance system of claim 3 where the track summary information is formatted as an extensible markup language (XML) document. ### Claim 16 (depends on 2) The video surveillance system of claim 2 wherein the track further comprises track data information. ### Claim 17 (depends on 16) The video surveillance system of claim 16 where the track identification system applies the unique identifier assigned to the at least one track to the track data information associated with the at least one track. ### Claim 18 (depends on 16) The video surveillance system of claim 16 where the track data information further comprises trajectory information. ### Claim 19 (depends on 16) The video surveillance system of claim 16 where the track data information is formatted as an extensible markup language (XML) document. ### Claim 20 (depends on 2) The video surveillance system of claim 2 wherein the track further comprises background image information. ### Claim 21 (depends on 20) The video surveillance system of claim 20 where the track identification system applies the unique identifier assigned to the at least one track to the background image information associated with the at least one track. ### Claim 22 (depends on 20) The video surveillance system of claim 20 where the background image information is formatted as an extensible markup language (XML) document. ### Claim 23 (depends on 2) The video surveillance system of claim 2 wherein: the video surveillance application and control interface further comprises: a track creation criterion specification system for permitting a user to specify the track creation criterion which controls when a track is created. ### Claim 24 (depends on 2) The video surveillance system of claim 2 wherein the video analysis engine performs the following operation: creating tracks in dependence on the at least one specified criterion, wherein each track is assigned a unique identifier which is used to identify all information related to that specific track. ### Claim 25 (depends on 24) The video surveillance system of claim 24 wherein the at least one database and application system performs the following operation: saving tracks created when the specified criterion is satisfied using the unique identifiers assigned to each track. ### Claim 26 (depends on 2) The video surveillance system of claim 2 wherein the at least one database and application system further comprises: a database server to save the at least one track using the unique identifier. ### Claim 27 (depends on 26) The video surveillance system of claim 26 where the video surveillance application and control interface further comprises: a track query formulation application to formulate queries to locate tracks saved in the database server satisfying a user-specified criterion. ### Claim 28 (depends on 27) The video surveillance system of claim 27 where the database and application system further comprises: an event retrieval system to accept user-specified track location queries and to locate tracks satisfying the criterion contained in the user-specified queries. ### Claim 29 (depends on 28) The video surveillance system of claim 28 where the database and application system further comprises: a track identifier forwarding system to forward track identifiers satisfying the user-specified query to the user requesting the track identifiers. ### Claim 30 (depends on 29) The video surveillance system of claim 29 where the video surveillance application and control interface further comprises: a track information request application to request track information associated with forwarded track identifiers. ### Claim 31 (depends on 30) The video surveillance system of claim 30 where the at least one database and application system further comprises: a track information retrieval application to retrieve track information associated with track identifiers. ### Claim 32 (depends on 31) The video surveillance system of claim 31 where the at least one database and application system further comprises: a transformation application for transforming the track information into a desired output format. ### Claim 33 (depends on 32) The video surveillance system of claim 32 where the desired output format is an extensible markup language (XML) document. ### Claim 34 (depends on 32) The video surveillance system of claim 32 where the desired output format is an HTML document. ### Claim 35 (depends on 31) The video surveillance system of claim 31 where the at least one database and application system further comprises: a track information forwarding system for forwarding track information retrieved from the at least one database and application system to the video surveillance application and control interface. ### Claim 36 (depends on 30) The video surveillance system of claim 30 wherein the track information further comprises track summary information. ### Claim 37 (depends on 30) The video surveillance system of claim 30 wherein the track information further comprises timestamp information memorializing a creation time of the track. ### Claim 38 (depends on 30) The video surveillance system of claim 30 wherein the track information further comprises object class information specifying an object class. ### Claim 39 (depends on 38) The video surveillance system of claim 38 where the object class corresponds to a vehicle class. ### Claim 40 (depends on 38) The video surveillance system of claim 38 where the object class corresponds to a human class. ### Claim 41 (depends on 30) The video surveillance system of claim 30 where the track information further comprises object area information. ### Claim 42 (depends on 30) The video surveillance system of claim 30 where the track information further comprises object velocity information. ### Claim 43 (depends on 30) The video surveillance system of claim 30 where the track information further comprises activity class information. ### Claim 44 (depends on 43) The video surveillance system of claim 43 where the activity class information corresponds to loitering activity. ### Claim 45 (depends on 30) The video surveillance system of claim 30 where the track information further comprises a pointer to a video segment. ### Claim 46 (depends on 30) The video surveillance system of claim 30 where the track information further comprises a pointer to a keyframe. ### Claim 47 (depends on 30) The video surveillance system of claim 30 wherein the track information further comprises track data information. ### Claim 48 (depends on 30) The video surveillance system of claim 30 where the track information further comprises trajectory information. ### Claim 49 (depends on 30) The video surveillance system of claim 30 wherein the track information further comprises background image information. ### Claim 50 (depends on 2) The video surveillance system of claim 2 where the video surveillance system further comprises: a local storage database associated with the at least one video surveillance system module and coupled to the at least one video surveillance system network for storing data created by the video analysis engine. ### Claim 51 (depends on 50) The video surveillance system of claim 50 further comprising: an off-peak forwarding application to forward data stored in the local storage database to the at least one database and application system during off-peak periods. ### Claim 52 (depends on 1) The video surveillance system of claim 1 where the at least one database and application system further comprises: an alert system, wherein the alert system further comprises: an alert criterion specification system to specify an alert based on data created by the video analysis engine; and an alert action specification system to specify alert actions to be taken when the alert criterion is satisfied. ### Claim 53 (depends on 52) The video surveillance system of claim 52 where the alert criterion concerns vehicle speed. ### Claim 54 (depends on 52) The video surveillance system of claim 52 where the alert criterion concerns loitering activity. ### Claim 55 (depends on 52) The video surveillance system of claim 52 where the alert action comprises an e-mail issued to a video surveillance analyst. ### Claim 56 (depends on 52) The video surveillance system of claim 52 where the alert action comprises a browser page comprising alert information. ### Claim 57 (depends on 56) The video surveillance system of claim 56 where the alert information comprises a keyframe. ### Claim 58 (depends on 56) The video surveillance system of claim 56 where the alert information comprises track identifiers. ### Claim 59 (depends on 56) The video surveillance system of claim 56 where the alert information comprises a video segment capturing activity that satisfied the alert criterion. ### Claim 60 (depends on 1) The video surveillance system of claim 1 further comprising: a summary system to create statistical data based on video analysis data generated by the video analysis engine. ### Claim 61 (depends on 1) The video surveillance system of claim 1 where: the video surveillance camera has a multiple programmable view feature, the video surveillance camera further comprising: a view programming system for programming views of the video surveillance camera; and the video surveillance system control interface further comprises: a video surveillance camera view programming system to program views of the video surveillance camera by interacting with the view programming system of the video surveillance camera. ### Claim 62 (depends on 61) The video surveillance system of claim 61 where the video analysis engine further comprises: a video surveillance camera view selection system to select a particular view of the video surveillance camera having the multiple programmable view feature and to analyze video associated with the view. ### Claim 63 (depends on 62) The video surveillance system of claim 62 wherein: the video analysis engine further comprises: a track identification system for assigning a unique identifier to the track created by the video analysis engine where the unique identifier references at least the selected view of the video surveillance camera. ### Claim 64 (depends on 63) The video surveillance system of claim 63 where the at least one database and application system performs the following operation: saving the track to a database server of the at least one database and application system. ### Claim 65 (depends on 64) The video surveillance system of claim 64 where the at least one database and application system performs the following operation: performing a query of the database server for tracks, wherein the query seeks tracks associated with the selected view of the video surveillance camera. ### Claim 66 (depends on 1) The video surveillance system of claim 1 wherein the at least one database and application system performs the following operation: saving the at least one tracks generated when the at least one user-specified video analysis criterion is satisfied. ### Claim 67 (depends on 66) The video surveillance system of claim 66 wherein the video surveillance system application and control interface performs the following operations: receiving a query for tracks corresponding to the at least one user-specified video analysis criterion; and transmitting the query for the tracks to the at least one database and application system. ### Claim 68 (depends on 67) The video surveillance system of claim 67 wherein the at least one database and application system performs the following operations: receiving the query for the tracks corresponding to the specified criterion; and locating tracks corresponding to the at least one user-specified video analysis criterion. ### Claim 69 (depends on 68) The video surveillance system of claim 68 wherein the at least one database and application system performs the following operation: transmitting the tracks satisfying the at least one user-specified video analysis criterion to the video surveillance system application and control interface. ### Claim 70 (depends on 69) The video surveillance system of claim 69 wherein the video surveillance system application and control interface performs the following operation: displaying the tracks received from the at least one database on the display. ### Claim 71 (depends on 68) The video surveillance system of claim 68 wherein the at least one database and application system performs the following operations: locating the video corresponding to the tracks; and transmitting the video satisfying the at least one user-specified video analysis criterion to the video surveillance system application and control interface. ### Claim 72 (depends on 71) The video surveillance system of claim 71 wherein the video surveillance application and control interface performs the following operations: displaying the video satisfying the at least one user-specified video analysis criterion. ### Claim 73 (depends on 2) The video surveillance system of claim 2 wherein the video surveillance system module further comprises: a keyframe generation system to create a keyframe of a specific field of view viewed by the video surveillance camera satisfying the specified video analysis criterion. ### Claim 74 (depends on 73) The video surveillance system of claim 73 where the keyframe generation system further comprises: a keyframe zoom system for creating a keyframe comprising an enlarged view of an object in the field of view of the video surveillance camera. ### Claim 75 (depends on 73) The video surveillance system of claim 73 where the track further comprise a pointer to the keyframe. ### Claim 76 (depends on 75) The video surveillance system of claim 75 wherein the keyframe generation system performs the following operation: generating a keyframe satisfying the at least one specified video analysis criterion; and the video analysis engine performs the following operation: creating tracks in dependence on the at least one specified video analysis criterion. ### Claim 77 (depends on 76) The video surveillance system of claim 76 wherein the at least one database and application system performs the following operation: saving the keyframe and track created when the specified criterion is satisfied. ### Claim 78 (depends on 77) The video surveillance system of claim 77 wherein the video surveillance application and control interface performs the following operations: receiving a query for tracks corresponding to the specified criterion; and transmitting the query for the tracks to the at least one database and application system. ### Claim 79 (depends on 78) The video surveillance system of claim 78 wherein the at least one database and application system performs the following operations: receiving the query for tracks corresponding to the specified criterion; and locating tracks corresponding to the specified criterion. ### Claim 80 (depends on 79) The video surveillance system of claim 79 wherein the at least one database and application system performs the following operations: locating the keyframe corresponding to the specified criterion by using the tracks; and transmitting the keyframe satisfying the specified criterion to the video surveillance application and control interface. ### Claim 81 (depends on 80) The video surveillance system of claim 80 wherein the video surveillance application and control interface performs the following operations: displaying the keyframe satisfying the specified criterion. ### Claim 82 (depends on 81) The video surveillance system of claim 81 wherein the video surveillance application and control interface further comprises: a keyframe-initiated video request system for requesting video corresponding to the specified criterion when the keyframe is of interest to a user. ### Claim 83 (depends on 82) The video surveillance system of claim 82 wherein the video surveillance application and control interface performs the following operations: receiving a request for video corresponding to the specified criterion from the keyframe-initiated video request system; and transmitting the request to the at least one database. ### Claim 84 (depends on 83) The video surveillance system of claim 83 wherein the at least one database and application system performs the following operations: locating the video corresponding to the specified criterion; and transmitting the video to the video surveillance application and control interface. ### Claim 85 (depends on 84) The video surveillance system of claim 84 wherein the video surveillance application and control interface performs the following operation: displaying the video corresponding to the specified criterion on the display. ### Claim 86 (depends on 2) The video surveillance system of claim 2 wherein the video analysis criterion corresponds to an object appearing in the video gathered by the video surveillance camera. ### Claim 87 (depends on 86) The video surveillance system of claim 86 wherein the video analysis engine performs the following operation: creating class data by identifying and categorizing an object appearing in the video gathered by the video surveillance camera. ### Claim 88 (depends on 86) The video surveillance system of claim 86 wherein the video analysis engine performs the following operation: creating a track for the object appearing in the video gathered by the video surveillance camera. ### Claim 89 (depends on 88) The video surveillance system of claim 88 wherein the track further comprises time stamp data identifying the time the object appeared in the video gathered by the video surveillance camera. ### Claim 90 (depends on 88) The video surveillance system of claim 88 wherein the track further comprises mask data for specifying locations of the object in frames of the video gathered by the video surveillance camera. ### Claim 91 (depends on 2) The video surveillance system of claim 2 wherein the video analysis criterion corresponds to an event appearing in video gathered by the video surveillance camera. ### Claim 92 (depends on 2) The video surveillance system of claim 2 wherein the video analysis criterion corresponds to an activity appearing in video gathered by the video surveillance camera. ### Claim 93 (depends on 1) The video surveillance system of claim 1 wherein: the video analysis engine further comprises: a data formatting system to accept data created by the video analysis engine and to format the data in machine readable data documents. ### Claim 94 (depends on 93) The video surveillance system of claim 93 wherein: the at least one database and application system further comprises: a data document transcription system to recover data stored in the machine readable data documents generated by the data formatting system and to convert the data to at least one format used by the database and application system. ### Claim 95 (depends on 1) The video surveillance system of claim 1 wherein the video surveillance system module further comprises: a video identification system to assign unique identification information to video gathered by the video surveillance camera; and wherein the video analysis engine further comprises: a data association system to associate data created by analyzing the video gathered by the video surveillance camera with the video using the unique identification information.