# US 2020/0401853 A1 · Smart video surveillance system using a neural network engine

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0401853 A1
- Document ID: US20200401853A1
- Status: Published application
- Kind: A1
- Application number: 16/446,316
- Filed: 2019-06-19
- Published: 2020-12-24
- Earliest claimed: 2019-06-19
- Assignees: Western Digital Technologies, Inc.
- Applicants: Western Digital Technologies, Inc.
- Inventors: Shaomin Xiong; Toshiki Hirano; Haoyu Wu
- Web page: https://patentgrep.com/patent/US20200401853A1

## CPC

- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/776: Validation; Performance evaluation
- G06V 10/82: using neural networks
- G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- H04N 7/181: {for receiving images from a plurality of remote sources}
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30232: Surveillance
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Remote neural network retraining for surveillance systems is disclosed. In some cases, systems and methods enable remote training and retraining of neural networks, while processing data in real life using the trained and retrained neural networks locally at the surveillance system. The surveillance system can determine a change of location and can retrain the neural network and/or initiate the retraining of the neural network remotely, such as on a cloud server, to retrain the neural network based on new image and/or video data taken from the new location. The remote server can transmit the retrained neural network and/or weights for nodes of the retrained neural network back to the surveillance system. The surveillance system can then update its neural network and process future image and/or video data based on the retrained neural network and/or weights.

## Claims

### Claim 1

A method of performing neural network computations in a surveillance system, the method comprising:
storing a plurality of weights of a neural network in a memory device for processing images or video from at least one image or video capture device;
performing a first inference operation locally at the surveillance system on data received from the at least one image or video capture device without transmitting the received data to a remote server as part of the first inference operation;
determining a change in location for the at least one image or video capture device;
transmitting, to a remove server, first image or video data received from the at least one image or video capture device at the changed location;
receiving, from the remote server, updated weights for the neural network responsive to the change in the location, the updated weights reflective of retraining of the neural network based on the first image or video data at the changed location;
receiving second image or video data from the at least one image or video capture device at the changed location; and
performing a second inference operation locally at the surveillance system on the second image or video data by processing the second image or video data in the neural network using the updated weights,
wherein the method is performed by one or more hardware processors.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first inference operation is performed locally at the surveillance system based on pre-configured weights.

### Claim 3 (depends on 1)

The method of claim 1, wherein determining the change in the location is based on user input.

### Claim 4 (depends on 1)

The method of claim 1, wherein determining the change in the location is based on data received from a positioning system device of the surveillance system.

### Claim 5 (depends on 1)

The method of claim 1, wherein the change in the location is automatically determined based on a determination of a change in a background of the first image or video data.

### Claim 6 (depends on 1)

The method of claim 1, wherein retraining the neural network is further based on an indication of a change in an inference operation type of the neural network.

### Claim 7 (depends on 6)

The method of claim 6, wherein the change of the inference operation type includes at least one of: a change of object to be detected or a change of an angle view of the at least one image or video capture device.

### Claim 8 (depends on 1)

The method of claim 1, wherein retraining the neural network is based on a real-time video stream from the at least one image or video capture device.

### Claim 9 (depends on 1)

The method of claim 1, wherein retraining the neural network is performed without an indication of an object present in the first image or video data.

### Claim 10

A surveillance system, the system comprising one or more hardware processors configured to:
store a plurality of weights of a neural network in a memory device for processing data from at least one sensing device;
perform a first inference operation on data received from the at least one sensing device without transmitting the received data to a server as part of the first inference operation;
determine a change in location for the at least one sensing device;
transmit, to a server, first sensed data received from the at least one sensing device at the changed location;
receive, from the server, updated weights for the neural network responsive to the change in the location, the updated weights reflective of re-training of the neural network based on the first sensed data at the changed location;
receive second sensed data from the at least one sensing device at the changed location; and
perform a second inference operation on the second sensed data by processing the second sensed data in the neural network using the updated weights.

### Claim 11 (depends on 10)

The system of claim 10, wherein the first sensed data comprises image or video data.

### Claim 12 (depends on 10)

The system of claim 10, wherein the one or more hardware processors are configured to store a plurality of preconfigured weights and the first inference operation is performed based on those weights.

### Claim 13 (depends on 10)

The system of claim 10, wherein the at least one sensing device is configured to transmit the first sensed data to the one or more hardware processors via wireless communication.

### Claim 14 (depends on 10)

The system of claim 10, wherein the second inference operation includes identifying an age or gender of a human in the second sensed data.

### Claim 15 (depends on 10)

The system of claim 10, wherein the at least one sensing device comprises at least one image or video capture device and wherein the change in location includes at least one of: a change in view angle of at least one of the at least one image or video capture device or a change in location outside of a previous view for the at least one image or video capture device.

### Claim 16

A system for performing neural network computations for surveillance, the system comprising:
means for storing a plurality of weights of a neural network into a memory device for processing images or video from at least one image or video capture device;
means for determining a change in location for the at least one image or video capture device;
means for retraining the neural network;
means for loading updated weights for the neural network;
means for receiving first image or video data from the at least one image or video capture device at the changed location responsive to the change in the location; and
means for performing an inference operation on the first image or video data by processing the first image or video data in the neural network using the updated weights.

### Claim 17 (depends on 16)

The system of claim 16, wherein the means for retraining the neural network is further for retrieving weights for another preconfigured neural network stored in the memory device.

### Claim 18 (depends on 16)

The system of claim 16, wherein the means for retraining the neural network is further for retrieving weights for another preconfigured neural network stored in a remote server.

### Claim 19 (depends on 16)

The system of claim 16, wherein the means for retraining the neural network is further for retraining based on user inputted data.

### Claim 20 (depends on 16)

The system of claim 16, wherein the system further comprises:
means for storing second image or video data received from the at least one image or video capture device at the changed location, wherein means for retraining the neural network is further for generating the updated weights based on the second image or video data at the changed location.