# US 2023/0056418 A1 · Video surveillance with neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0056418 A1
- Document ID: US20230056418A1
- Status: Published application
- Kind: A1
- Application number: 17/896,856
- Filed: 2022-08-26
- Published: 2023-02-23
- Earliest claimed: 2018-07-05
- Assignees: Movidius Limited
- Applicants: Movidius Limited
- Inventors: David Moloney
- Web page: https://patentgrep.com/patent/US20230056418A1

## CPC

- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/82: using neural networks
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G08B 13/19615: {wherein said pattern is defined by the user}
- G08B 13/19671: {Addition of non-video data, i.e. metadata, to video stream}
- G08B 13/19682: {Graphic User Interface [GUI] presenting system data to the user, e.g. information on a screen helping a user interacting with an alarm system}
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 20/44: {Event detection}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS

## Abstract

Example methods, apparatus, systems and articles of manufacture (e.g., physical storage media) to implement video surveillance with neural networks are disclosed. Example systems disclosed herein include a database to store records of operator-labeled video segments (e.g., as records of operator-labeled video segments). The operator-labeled video segments include reference video segments and corresponding reference event labels describing the video segments. Disclosed example systems also include a neural network including a first instance of an inference engine, and a training engine to train the first instance of the inference engine based on a training set of the operator-labeled video segments obtained from the database, the first instance of the inference engine to infer events from the operator-labeled video segments included in the training set. Disclosed example systems further include a second instance of the inference engine to infer events from monitored video feeds, the second instance of the inference engine being based on the first instance of the inference engine.

## Claims

### Claim 1

A video surveillance system comprising:
a database to store operator-labeled video segments, the operator-labeled video segments including reference video segments and corresponding reference event labels to describe the reference video segments;
a neural network including:
a first instance of an inference engine; and
a training engine to train the first instance of the inference engine based on a training set of the operator-labeled video segments obtained from the database, the first instance of the inference engine to infer events from the operator-labeled video segments included in the training set; and
a second instance of the inference engine to infer events from monitored video feeds, the second instance of the inference engine being based on the first instance of the inference engine.

### Claim 2 (depends on 1)

The video surveillance system of claim 1, wherein the reference event labels indicate whether the corresponding reference video segments depict a defined event.

### Claim 3 (depends on 1)

The video surveillance system of claim 1, wherein a first one of the reference event labels corresponding to a first one of the reference video segments indicates (i) a type of event and (ii) whether the first one of the reference video segments depicts the type of event.

### Claim 4 (depends on 1)

The video surveillance system of claim 1, further including a monitoring station, the monitoring station including:
a display to present a first one of the monitored video feeds; and
a monitoring interface to generate operator event labels based on operator decisions corresponding to monitored video segments of the first one of the monitored video feeds.

### Claim 5 (depends on 4)

The video surveillance system of claim 4, wherein the database is in communication with the monitoring station to receive a first one of the monitored video segments and a first one of the operator event labels corresponding to the first one of the monitored video segments, the database to store the first one of the monitored video segments and the corresponding first one of the operator event labels as a first one of the reference video segments and a corresponding first one of the reference event labels included in a first one of the operator-labeled video segments.

### Claim 6 (depends on 5)

The video surveillance system of claim 5, wherein the monitoring station is further to implement the second instance of the inference engine.

### Claim 7 (depends on 6)

The video surveillance system of claim 6, wherein the second instance of the inference engine is to output an inferred event for a second one of monitored video segments of the first one of the monitored video feeds, the monitoring interface is to generate a second one of the operator event labels from an operator decision detected for the second one of monitored video segments, and the monitoring station further includes a comparator to compare the inferred event and the second one of the operator event labels to obtain updated training data.

### Claim 8 (depends on 7)

The video surveillance system of claim 7, wherein the neural network is in communication with the monitoring station to receive the updated training data, and the training engine is to retrain the first instance of the inference engine based on the updated training data.

### Claim 9

At least one non-transitory computer readable storage media comprising computer readable instructions which, when executed, cause one or more processors to at least:
train a first instance of an inference engine based on a training set of operator-labeled video segments, the operator-labeled video segments including reference video segments and corresponding reference event labels to describe the reference video segments, the first instance of the inference engine to infer events from the operator-labeled video segments included in the training set; and
deploy a second instance of the inference engine to infer events from monitored video feeds, the second instance of the inference engine being based on the first instance of the inference engine.

### Claim 10 (depends on 9)

The at least one storage media of claim 9, wherein the reference event labels indicate whether the corresponding reference video segments depict a defined event.

### Claim 11 (depends on 9)

The at least one storage media of claim 9, wherein a first one of the reference event labels corresponding to a first one of the reference video segments indicates (i) a type of event and (ii) whether the first one of the reference video segments depicts the type of event.

### Claim 12 (depends on 9)

The at least one storage media of claim 9, wherein the computer readable instructions, when executed, cause the one or more processors to obtain a first one of the reference video segments and a corresponding first one of the reference event labels from a monitoring station.

### Claim 13 (depends on 12)

The at least one storage media of claim 12, wherein the computer readable instructions, when executed, cause the one or more processors to deploy the second instance of the inference engine to the monitoring station.

### Claim 14 (depends on 13)

The at least one storage media of claim 13, wherein the second instance of the inference engine is a duplicate of the first instance of the inference engine when the second instance of the inference engine is initially deployed to the monitoring station.

### Claim 15 (depends on 13)

The at least one storage media of claim 13, wherein the monitoring station is to obtain updated training data by comparing (i) inferred events output by the second instance of the inference engine for segments of the monitored video feeds with (ii) operator event labels generated by the monitoring station for the segments of the monitored video feeds, and the computer readable instructions, when executed, further cause the one or more processors to retrain the first instance of the inference engine based on the updated training data.

### Claim 16

An apparatus comprising:
means for obtaining operator-labeled video segments, the operator-labeled video segments including reference video segments and corresponding reference event labels to describe the reference video segments;
means for training a first instance of an inference engine based on a training set of the operator-labeled video segments, the first instance of the inference engine to infer events from the operator-labeled video segments included in the training set; and
means for deploying a second instance of the inference engine to infer events from monitored video feeds, the second instance of the inference engine being based on the first instance of the inference engine.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein the reference event labels indicate whether the corresponding reference video segments depict a defined event.

### Claim 18 (depends on 16)

The apparatus of claim 16, wherein a first one of the reference event labels corresponding to a first one of the reference video segments indicates (i) a type of event and (ii) whether the first one of the reference video segments depicts the type of event.

### Claim 19 (depends on 16)

The apparatus of claim 16, wherein the means for obtaining the records of operator-labeled video segments is to obtain a first one of the reference video segments and a corresponding first one of the reference event labels from a monitoring station.

### Claim 20 (depends on 19)

The apparatus of claim 19, wherein the means for deploying the second instance of the inference engine is to deploy the second instance of the inference engine to the monitoring station.

### Claim at position 21

21-29. (canceled)