# US 11,308,335 B2 · Intelligent video surveillance system and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,335 B2
- Document ID: US11308335B2
- Status: Granted
- Kind: B2
- Application number: 16/876,535
- Filed: 2020-05-18
- Published: 2020-11-19
- Granted: 2022-04-19
- Earliest claimed: 2019-05-17
- Assignees: ZeroEyes, Inc.
- Applicants: ZeroEyes, Inc.
- Inventors: Timothy Sulzer; Michael Lahiff; Marcus Day
- Web page: https://patentgrep.com/patent/US11308335B2

## CPC

- H04N 7/18: Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/2413: based on distances to training or reference patterns
- G06F 18/28: Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- H04N 7/183: {for receiving images from a single remote source}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An intelligent video surveillance system is disclosed which performs real-time analytics on a live video stream. The system includes a training database populated with frames of actual video of objects of interest taken in a relevant environment. A subset of the frames include bounding boxes and/or bounding polygons which can be augmented. The training database also includes classification/annotation of data/labels relevant to the object of interest, a person carrying the object of interest, and/or the background or environment. The training database is searchable by the classification/annotation of data/labels.

## Claims

### Claim 1

A method for training an object detection device, the method comprising the steps of:
recording a first video stream from a security camera;
selecting a set of frames from the first video stream;
selecting a subset of frames from the set of frames and inserting a bounding box in each frame of the subset of frames;
creating a list of attributes associated with said each frame of the subset of frames;
creating a database comprising said each frame of the subset of frames and the associated list of attributes, wherein the database is searchable by at least one attribute of the list of attributes;
training a detection model using the database, wherein the training includes varying a first parameter of the detection model;
analyzing a second video stream using the detection model to determine a number of true positive events and a number of false positive events, wherein the second video stream is different from the first video stream;
creating a dataset of the analysis of the second video stream;
filtering the dataset using a second parameter;
determining a ratio of true positive events to false positive events (“TP/FP”);
determining a ratio of false positive events to true positive events (“FP/TP”);
converting the FP/TP to a percentage (“% FP/TP); and
evaluating a performance of the detection model based at least in part on a ratio of % FP/TP to TP/FP.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first video stream comprises an image of a person carrying an object of interest.

### Claim 3 (depends on 1)

The method of claim 1, wherein the object of interest is a weapon.

### Claim 4 (depends on 1)

The method of claim 1, wherein at least one frame of the subset of frames includes an object of interest.

### Claim 5 (depends on 4)

The method of claim 4, wherein the bounding box circumscribes at least a part of the object of interest.

### Claim 6 (depends on 1)

The method of claim 1, wherein the bounding box is a bounding polygon.

### Claim 7 (depends on 1)

The method of claim 1, wherein at least one of the attributes is associated with an object of interest.

### Claim 8 (depends on 1)

The method of claim 1, wherein the attributes are selected from the group consisting of: color, lighting level, clarity, security camera characteristics, an object of interest type, an orientation of the object of interest, contrast, and combinations thereof.

### Claim 9 (depends on 1)

The method of claim 1 further comprising the step of adjusting a location of the bounding box in one of the frames of the subset of frames from a first location to a second location.

### Claim 10 (depends on 9)

The method of claim 9, wherein the first location and the second location are separated by a predetermined number of pixels.

### Claim 11 (depends on 1)

The method of claim 1, further comprising the step of adjusting a size of the bounding box in one of the frames of the subset of frames from a first size to a second size.

### Claim 12 (depends on 11)

The method of claim 11, wherein the first size of the bounding box comprises a first number of pixels and the second size of the bounding box comprises a second number of pixels different from the first number of pixels.

### Claim 13 (depends on 1)

The method of claim 1, wherein the first parameter is selected from the group consisting of: a number of labels, a number of images, a number of frames, a number of iterations, a max iteration value, a test iteration value, a test interval, a momentum value, a ratio value, a learning rate, a weight decay, and combinations thereof.

### Claim 14 (depends on 1)

The method of claim 1, wherein the second video stream comprises a live video stream or a filtered video stream.

### Claim 15 (depends on 1)

The method of claim 1, wherein the second parameter is selected from the group consisting of: confidence score, event duration, pixel area size, object speed, minimum range of object movement, average object size, and average pixel speed, and combinations thereof.

### Claim 16 (depends on 1)

The method of claim 1, further comprising the step of maximizing a difference between the number of true positive events and the number of false positive events, wherein the number of true positive events is greater than the number of false positive events.

### Claim 17 (depends on 1)

The method of claim 1, wherein the evaluation of the performance of the detection model comprises determining performance metrics.

### Claim 18 (depends on 17)

The method of claim 17, wherein the performance metrics are selected from the group consisting of: number of true positive events, number of false positive events, a score value, an average score value, a label performance value, a score by distance value, and combinations thereof.

### Claim 19 (depends on 1)

The method of claim 1, further comprising the step of comparing the evaluated performance of the detection model with a performance of a second detection model.

### Claim 20 (depends on 1)

The method of claim 1, further comprising the step of adjusting a location of the insertion of the bounding box in a frame of the subset of frames based on the evaluated performance of the detection model.

### Claim 21 (depends on 1)

The method of claim 1, further comprising the step of changing an attribute in the list of attributes based on the evaluated performance of the detection model.

### Claim 22 (depends on 1)

The method of claim 1, further comprising the step of adding or deleting an attribute from the list of attributes based on the evaluated performance of the detection model.

### Claim 23

A system for training an object detection device, comprising:
a recording device for recording a first video stream from a security camera;
a processor operatively connected to said recording device, said processor programmed to select a set of frames from the first video stream;
said processor further programmed to select a subset of frames from the set of frames and inserting a bounding box in each frame of the subset of frames;
said processor further programmed to create a list of attributes associated with said each frame of the subset of frames;
a database operatively connected to said processor, wherein said processor populates said database such that said database comprises said each frame of the subset of frames and the associated list of attributes, wherein said database is searchable by at least one attribute of the list of attributes;
said processor further programmed to train a detection model using the database, wherein the training includes varying a first parameter of the detection model;
said processor further programmed to analyze a second video stream using the detection model to determine a number of true positive events and a number of false positive events, wherein the second video stream is different from the first video stream;
said processor further programmed to create a dataset of the analysis of the second video stream;
said processor further programmed to filter the dataset using a second parameter;
said processor further programmed to evaluate a performance of the detection model, at least in part, by:
determining a ratio of true positive events to false positive events (“TP/FP”),
determining a ratio of false positive events to true positive events (“FP/TP”),
converting the FP/TP to a percentage (“% FP/TP), and
determining a ratio of % FP/TP to TP/FP.

### Claim 24

A non-transitory, machine-readable medium having stored thereon a plurality of
executable instructions, the plurality of executable instructions comprising instructions to:
record a first video stream from a security camera;
select a set of frames from the first video stream;
select a subset of frames from the set of frames and inserting a bounding box in each frame of the subset of frames;
create a list of attributes associated with said each frame of the subset of frames;
create a database comprising said each frame of the subset of frames and the associated list of attributes, wherein the database is searchable by at least one attribute of the list of attributes;
train a detection model using the database, wherein the training includes varying a first parameter of the detection model;
analyze a second video stream using the detection model to determine a number of true positive events and a number of false positive events, wherein the second video stream is different from the first video stream;
create a dataset of the analysis of the second video stream;
filter the dataset using a second parameter
determine a ratio of true positive events to false positive events (“TP/FP”);
determine a ratio of false positive events to true positive events (“FP/TP”);
convert the FP/TP to a percentage (“% FP/TP); and
evaluate a performance of the detection model based at least in part on a ratio of % FP/TP to TP/FP.