# US 12,008,794 B2 · Systems and methods for intelligent video surveillance > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,794 B2 - Document ID: US12008794B2 - Status: Granted - Kind: B2 - Application number: 17/239,536 - Filed: 2021-04-23 - Published: 2021-08-05 - Granted: 2024-06-11 - Earliest claimed: 2018-10-25 - Assignees: Shanghai Truthvision Information Technology Co., Ltd. - Applicants: Shanghai Truthvision Information Technology Co., Ltd. - Inventors: Zhong Zhang - Web page: https://patentgrep.com/patent/US12008794B2 ## CPC - G06V 10/255: {Detecting or recognising potential candidate objects based on visual cues, e.g. shapes} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 7/292: Multi-camera tracking - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - H04N 7/188: {Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position (signal generation from motion picture films H04N5/253)} - G06T 2207/20076: Probabilistic image processing - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30196: Human being; Person - G06T 2207/30232: Surveillance - G06T 2207/30241: Trajectory - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method may include obtaining a video collected by a visual sensor, the video including a plurality of frames and detecting one or more objects from the video in at least a portion of the plurality of frames. The method may also include determining a first detection result associated with the one or more objects with a trained self-learning model. The method may further include selecting a target moving object of interest from the one or more objects at least in part based on the first detection result. The trained self-learning model may be provided based on a plurality of training samples collected by the visual sensor. ## Claims ### Claim 1 A system, comprising: a storage device storing a set of instructions; and at least one processor configured to communicate with the storage device, wherein when executing the set of instructions, the at least one processor is directed to cause the system to perform operations including: obtaining a video collected by a visual sensor, the video including a plurality of frames; detecting, in at least a portion of the plurality of frames, one or more objects from the video; determining, with a trained self-learning model, a first detection result associated with the one or more objects; determining, based on the at least a portion of the plurality of frames, one or more behavior features associated with each of the one or more objects; determining, based on the one or more behavior features associated with each of the one or more objects, a second detection result associated with each of the one or more objects; and determining, based on the first detection result and the second detection result, a tartlet moving object of interest from the one or more objects, wherein the trained self-learning model is provided based on a plurality of training samples collected by the visual sensor. ### Claim 2 (depends on 1) The system of claim 1, wherein to detect, in at least a portion of the plurality of frames, one or more objects from the video, the at least one processor is directed to cause the system to perform the operations including: detecting the one or more objects from the video using an object detection model. ### Claim 3 (depends on 2) The system of claim 2, wherein the object detection model is constructed based on a deep learning model. ### Claim 4 (depends on 1) The system of claim 1, wherein to determine, based on the at least a portion of the plurality of frames, one or more behavior features associated with each of the one or more objects, the at least one processor is directed to cause the system to perform the operations including: determining, based on the at least a portion of the plurality of frames and a prior calibration model of the visual sensor determined in a last calibration, the one or more behavior features. ### Claim 5 (depends on 1) The system of claim 1, wherein the one or more behavior features of the one or more objects include at least one of a speed, an acceleration, a trajectory, a movement amplitude, a direction, a movement frequency, or voice information. ### Claim 6 (depends on 1) The system of claim 1, wherein the trained self-learning model is generated by a process including: obtaining the plurality of training samples each of which includes a historical video collected by the visual sensor; detecting one or more motion subjects from the historical video for each of the plurality of training samples; and training a self-learning model using information associated with the detected one or more motion subjects to obtain the trained self-learning model. ### Claim 7 (depends on 6) The system of claim 6, wherein the information associated with the detected one or more motion subjects includes at least one of: time information when the detected one or more motion subjects recorded by the historical video; spatial information associated with the detected one or more motion subjects; weather information when the detected one or more motion subjects recorded by the historical video; or motion information of the detected one or more motion subjects. ### Claim 8 (depends on 1) The system of claim 1, wherein the trained self-learning model includes a first part relating to reference knowledge of different scenes and a second part relating to learned knowledge generated from a training process of the trained self-learning model. ### Claim 9 (depends on 8) The system of claim 8, wherein the reference knowledge of different scenes includes characteristics of one or more subjects appearing in each of the different scenes. ### Claim 10 (depends on 1) The system of claim 1, wherein the first detection result includes one or more first candidate moving objects of interest and the second detection result includes one or more second candidate moving objects of interest, and to determine, based on the first detection result and the second detection result, the target moving object of interest from the one or more objects, the at least one processor is directed to cause the system to perform the operations including: designating a same candidate moving object of interest from the one or more first candidate moving objects of interest and the one or more second candidate moving objects of interest as the target moving object of interest. ### Claim 11 (depends on 1) The system of claim 1, wherein the first detection result includes a first probability that each of the one or more objects is a moving object of interest, the second detection result includes a second probability that each of the one or more objects is a moving object of interest, and to determine, based on the first detection result and the second detection result, the target moving object of interest from the one or more objects, the at least one processor is directed to cause the system to perform the operations including: designating a moving object having a first probability exceeding a first threshold and a second probability exceeding a second threshold as the target moving object of interest. ### Claim 12 (depends on 1) The system of claim 1, wherein the at least one processor is directed to cause the system to perform additional operations including: in response to a detection of the target moving object of interest from the video, generating feedback relating to the detection of the target moving object of interest; and transmitting the feedback relating to the detection of the target moving object of interest to a terminal. ### Claim 13 (depends on 12) The system of claim 12, wherein the feedback includes a notification indicating that a moving object exists. ### Claim 14 (depends on 1) The system of claim 1, wherein the at least one processor is directed to cause the system to perform additional operations including: in response to a detection of each of at least a portion of the one or more objects from the video, generating candidate feedbacks each of which relates to the detection of one of the one or more objects; determining, based on at least one of the first detection result or the second detection result, target feedback from the candidate feedbacks; and transmitting the target feedback to a terminal. ### Claim 15 (depends on 1) The system of claim 1, wherein the at least one processor is further configured to cause the system to perform additional operations including: determining, based on the target moving object of interest, a calibration model of the visual sensor, the calibration model describing a transform relationship between a two-dimensional (2D) coordinate system and a three-dimensional (3D) coordinate system of the visual sensor. ### Claim 16 (depends on 15) The system of claim 15, wherein to determine, based on the target moving object of interest, a calibration model of the visual sensor, the at least one processor is further configured to cause the system to perform the additional operations including: determining, based on the at least a portion of the plurality of frames, an estimated value of a characteristic of the target moving object of interest denoted by the 2D coordinate system; and determining, based on the estimated value and a reference value of the characteristic of the target moving object of interest denoted by the 3D coordinate system, the calibration model. ### Claim 17 (depends on 16) The system of claim 16, wherein the characteristic of the target moving object of interest includes a physical size of at least a portion of the target moving object of interest. ### Claim 18 (depends on 1) The system of claim 1, wherein the target moving object of interest includes at least one of a person, a vehicle, or an animal whose motion includes an anomaly. ### Claim 19 A method implemented on a computing device having at least one processor and at least one computer-readable storage medium for abnormal scene detection, the method comprising: obtaining a video collected by a visual sensor, the video including a plurality of frames; detecting, in at least a portion of the plurality of frames, one or more objects from the video; determining, with a trained self-learning model, a first detection result associated with the one or more objects; determining, based on the at least a portion of the plurality of frames, one or more behavior features associated with each of the one or more objects; determining, based on the one or more behavior features associated with each of the one or more objects, a second detection result associated with each of the one or more objects; and determining, based on the first detection result and the second detection result, a target moving object of interest from the one or more objects, wherein the trained self-learning model is provided based on a plurality of training samples collected by the visual sensor. ### Claim 20 A non-transitory computer readable medium, comprising: instructions being executed by at least one processor, causing the at least one processor to implement a method, comprising: obtaining a video collected by a visual sensor, the video including a plurality of frames; detecting, in at least a portion of the plurality of frames, one or more objects from the video; determining, with a trained self-learning model, a first detection result associated with the one or more objects; determining, based on the at least a portion of the plurality of frames, one or more behavior features associated with each of the one or more objects; determining, based on the one or more behavior features associated with each of the one or more objects, a second detection result associated with each of the one or more objects; and determining, based on the first detection result and the second detection result, a target moving object of interest from the one or more objects, wherein the trained self-learning model is provided based on a plurality of training samples collected by the visual sensor.