# US 2024/0203127 A1 · Dynamic edge-cloud collaboration with knowledge adaptation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0203127 A1
- Document ID: US20240203127A1
- Status: Published application
- Kind: A1
- Application number: 18/554,461
- Filed: 2022-04-06
- Published: 2024-06-20
- Earliest claimed: 2021-04-06
- Assignees: Wyze Labs, Inc.
- Applicants: Wyze Labs, Inc.
- Inventors: Mohammadmahdi Kamani; Lin Chen; Zhongwei Cheng; Tianqiang Liu
- Web page: https://patentgrep.com/patent/US20240203127A1

## CPC

- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06F 16/55: Clustering; Classification
- G06N 3/045: Combinations of networks
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/09: Supervised learning
- G06N 3/096: Transfer learning
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/82: using neural networks
- G06V 10/95: {structured as a network, e.g. client-server architectures}
- H04N 7/183: {for receiving images from a single remote source}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Introduced here are different variations of an edge-cloud collaboration framework (also called an “ECC framework”) that learns models with different levels of tradeoffs between the aforementioned objectives that tend to conflict with one another. This ECC framework—based on an adaptation of knowledge from “edge models” employed by edge devices to “cloud models” employed by a computer server system—can attempt to minimize the communication and computation costs during the inference stage while also trying to achieve the best performance possible.

## Claims

### Claim 1

A surveillance system comprising:
a camera that is configured to:
generate a first series of images of an environment to be surveilled,
apply a first model to each image in the first series of images, so as to produce a first series of outputs, and
for each output in the first series of outputs,
determine whether confidence in that output exceeds a threshold, and
cause transmission of that output to a server system in response to a determination that the confidence does not exceed the threshold; and
the server system that is configured to:
receive a second series of images from the camera,
wherein the second series of images is representative of a subset of the first series of images, and
apply a second model to each image in the second series of images, so as to produce a second series of outputs.

### Claim 2 (depends on 1)

The surveillance system of claim 1,
wherein each output in the first series of outputs is representative of an inference made by the first model regarding content of a corresponding image in the first series of images, and
wherein each output in the second series of outputs is representative of an inference made by the second model regarding content of a corresponding image in the second series of images.

### Claim 3 (depends on 1)

The surveillance system of claim 1, wherein the camera is further configured to:
for each output in the first series of outputs,
indicate that output is an appropriate inference in a data structure in response to a determination that the confidence does exceed the threshold.

### Claim 4 (depends on 1)

The surveillance system of claim 1,
wherein the server system is further configured to:
cause transmission of the second series of outputs to the camera; and
wherein the camera is further configured to:
establish that an activity or an object of interest is included in at least one image in the first series of images based on an analysis of (i) confident outputs in the first series of outputs and (ii) the second series of outputs, and
cause a notification that specifies the activity or the object of interest to be presented by a computer program executing on a mediatory device.

### Claim 5 (depends on 1)

The surveillance system of claim 1,
wherein the server system is further configured to:
cause transmission of the second series of outputs to a computer program executing on a mediatory device; and
wherein the camera is further configured to:
cause transmission of confident outputs in the first series of outputs to the computer program executing on the mediatory device.

### Claim 6 (depends on 1)

The surveillance system of claim 1, wherein the first model requires fewer computational resources than the second model to produce an output when applied to a given image.

### Claim 7 (depends on 1)

The surveillance system of claim 1, wherein the threshold is programmed in memory of the camera.

### Claim 8

A surveillance system comprising:
a camera that is configured to:
generate a series of images of an environment to be surveilled,
apply a first model to each image in the series of images, so as to produce a first series of outputs, and
for each output in the first series of outputs,
determine whether confidence in that output exceeds a threshold, and
cause transmission of a feature map corresponding to that output to a server system in response to a determination that the confidence does not exceed the threshold; and
the server system that is configured to:
receive a series of feature maps from the camera,
wherein the series of feature maps corresponds to a subset of the series of images, and
for each feature map of the series of feature maps,
providing that feature map as input to a second model, so as to produce a second series of outputs.

### Claim 9 (depends on 8)

The surveillance system of claim 8, wherein each feature map is provided as input to a middle layer of the second model.

### Claim 10 (depends on 8)

The surveillance system of claim 8, wherein the first and second models are classification models.

### Claim 11 (depends on 8)

The surveillance system of claim 8, wherein the first and second models are object detection models.

### Claim 12

A method performed by an edge device that generates samples while surveilling an environment, the method comprising:
applying a model to the samples to produce outputs,
wherein each output is representative of an inference made in relation to a corresponding sample;
determining whether confidence in each of the outputs exceeds a threshold; and
for each output for which the confidence does not exceed the threshold,
causing transmission of (i) the corresponding sample or (ii) information regarding the corresponding sample to a server system for analysis.

### Claim 13 (depends on 12)

The method of claim 12, wherein the edge device is a camera, and wherein the model is trained to detect instances of an object in images.

### Claim 14 (depends on 12)

The method of claim 12, wherein the information includes a feature map derived for the corresponding sample.

### Claim 15 (depends on 12)

The method of claim 12, wherein said applying, said determining, and said causing are performed in real time as the samples are generated by the edge device.

### Claim 16

A method performed by a server system, the method comprising:
receiving a feature map from an edge device that generates a sample while surveilling an environment,
wherein the feature map is generated by a first model upon being applied to the sample;
providing the feature map to an intermediate layer of a second model as input, so as to produce an output that is representative of an inference made in relation to the sample; and
storing an indication of the inference in a data structure.

### Claim 17 (depends on 16)

The method of claim 16, wherein the data structure is maintained in memory of the server system.

### Claim 18 (depends on 16)

The method of claim 16, wherein the sample is representative of a digital image of the environment.
