# US 12,001,513 B2 · Self-optimizing video analytics pipelines

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,001,513 B2
- Document ID: US12001513B2
- Status: Granted
- Kind: B2
- Application number: 17/522,226
- Filed: 2021-11-09
- Published: 2022-06-02
- Granted: 2024-06-04
- Earliest claimed: 2020-11-30
- Assignees: NEC Corporation
- Applicants: NEC Laboratories America, Inc.
- Inventors: Giuseppe Coviello; Yi Yang; Srimat Chakradhar
- Web page: https://patentgrep.com/patent/US12001513B2

## CPC

- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06F 9/5027: {the resource being a machine, e.g. CPUs, Servers, Terminals}
- G06F 9/5055: {considering software capabilities, i.e. software resources associated or available to the machine}
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06V 10/82: using neural networks
- G06V 10/94: Hardware or software architectures specially adapted for image or video understanding
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06F 2209/508: Monitor
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method for implementing a self-optimized video analytics pipeline is presented. The method includes decoding video files into a sequence of frames, extracting features of objects from one or more frames of the sequence of frames of the video files, employing an adaptive resource allocation component based on reinforcement learning (RL) to dynamically balance resource usage of different microservices included in the video analytics pipeline, employing an adaptive microservice parameter tuning component to balance accuracy and performance of a microservice of the different microservices, applying a graph-based filter to minimize redundant computations across the one or more frames of the sequence of frames, and applying a deep-learning-based filter to remove unnecessary computations resulting from mismatches between the different microservices in the video analytics pipeline.

## Claims

### Claim 1

A method for implementing a self-optimized video analytics pipeline, the method comprising:
decoding video files into a sequence of frames;
extracting features of objects from one or more frames of the sequence of frames of the video files;
employing an adaptive resource allocation component based on reinforcement learning (RL) to dynamically balance resource usage of different microservices included in the video analytics pipeline;
employing an adaptive microservice parameter tuning component to balance accuracy and performance of a microservice of the different microservices;
applying a graph-based filter to minimize redundant computations across the one or more frames of the sequence of frames; and
applying a deep-learning-based filter to remove unnecessary computations resulting from mismatches between the different microservices in the video analytics pipeline.

### Claim 2 (depends on 1)

The method of claim 1, wherein a state in the adaptive resource allocation component is a vector including at least backlog of input queue of each microservice, current utilization of system resources, a fraction of each system resource assigned to each of the microservices, and metrics derived from content of the video files.

### Claim 3 (depends on 1)

The method of claim 1, wherein a microservice of the different microservices includes a monolithic application having functions that are dis-aggregated into separate object detection and feature extraction microservices.

### Claim 4 (depends on 1)

The method of claim 1, wherein the adaptive microservice parameter tuning component tunes a batch size parameter and an image resolution parameter by RL.

### Claim 5 (depends on 1)

The method of claim 1, wherein the graph-based filter removes repeated objects from consecutive frames within the sequence of frames by creating a graph G=(V, E), where each object forms a vertex vi and an edge (vi, vj) between two vertices vi and vj, if and only if, these objects are close to each other, the closeness of two objects determined by spatial closeness and temporal closeness used to compute a distance metric.

### Claim 6 (depends on 5)

The method of claim 5, wherein the spatial closeness employs an overlap ratio between bounding boxes of the objects from the consecutive frames, the overlap ratio given as:
overlap
⁢
⁢
(
v
i
,
v
j
)
=
area
⁢
⁢
(
v
i
⋂
v
j
)
min
⁡
(
area
⁢
⁢
(
v
i
)
,
area
⁢
⁢
(
v
j
)
)
.

### Claim 7 (depends on 5)

The method of claim 5, wherein the temporal closeness employs a lag between the objects from the consecutive frames, the lag given as:
lag
⁢
⁢
(
v
i
,
v
j
)
=

frame
⁢
⁢
#
⁢
v
i
-
frame
⁢
⁢
#
⁢
v
j

frameInBatch
.

### Claim 8 (depends on 5)

The method of claim 5, wherein the graph is partitioned into multiple sub-graphs, each sub-graph is flattened into a list of objects, and duplicates are labeled in a flattened connected component.

### Claim 9

A non-transitory computer-readable storage medium comprising a computer-readable program for implementing a self-optimized video analytics pipeline, wherein the computer-readable program when executed on a computer causes the computer to perform the steps of;
decoding video files into a sequence of frames;
extracting features of objects from one or more frames of the sequence of frames of the video files;
employing an adaptive resource allocation component based on reinforcement learning (RL) to dynamically balance resource usage of different microservices included in the video analytics pipeline;
employing an adaptive microservice parameter tuning component to balance accuracy and performance of a microservice of the different microservices;
applying a graph-based filter to minimize redundant computations across the one or more frames of the sequence of frames; and
applying a deep-learning-based filter to remove unnecessary computations resulting from mismatches between the different microservices in the video analytics.

### Claim 10 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein a state in the adaptive resource allocation component is a vector including at least backlog of input queue of each microservice, current utilization of system resources, a fraction of each system resource assigned to each of the microservices, and metrics derived from content of the video files.

### Claim 11 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein a microservice of the different microservices includes a monolithic application having functions that are dis-aggregated into separate object detection and feature extraction microservices.

### Claim 12 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein the adaptive microservice parameter tuning component tunes a batch size parameter and an image resolution parameter by RL.

### Claim 13 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein the graph-based filter removes repeated objects from consecutive frames within the sequence of frames by creating a graph G=(V, E), where each object forms a vertex vi and an edge e(vi, vj) between two vertices vi and vj, if and only if, these objects arc close to each other, the closeness of two objects determined by spatial closeness and temporal closeness used to compute a distance metric.

### Claim 14 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, wherein the spatial closeness employs an overlap ratio between bounding boxes of the objects from the consecutive frames, the overlap ratio given as:
overlap
⁢
⁢
(
v
i
,
v
j
)
=
area
⁢
⁢
(
v
i
⋂
v
j
)
min
⁡
(
area
⁢
⁢
(
v
i
)
,
area
⁢
⁢
(
v
j
)
)
.

### Claim 15 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, wherein the temporal closeness employs a lag between the objects from the consecutive frames, the lag given as:
lag
⁢
⁢
(
v
i
,
v
j
)
=

frame
⁢
⁢
#
⁢
v
i
-
frame
⁢
⁢
#
⁢
v
j

frameInBatch
.

### Claim 16 (depends on 13)

The non-transitory computer-readable storage medium of claim 13, wherein the graph is partitioned into multiple sub-graphs, each sub-graph is flattened into a list of objects, and duplicates are labeled in a flattened connected component.

### Claim 17

A system for implementing a self-optimized video analytics pipeline, the system comprising:
a memory; and
one or more processors in communication with the memory configured to:
decode video files into a sequence of frames;
extract features of objects from one or more frames of the sequence of frames of the video files;
employ an adaptive resource allocation component based on reinforcement learning (RL) to dynamically balance resource usage of different microservices included in the video analytics pipeline;
employ an adaptive microservice parameter tuning, component to balance accuracy and performance of a microservice of the different microservices;
apply a graph-based filter to minimize redundant computations across the one or more frames of the sequence of frames; and
apply a deep-learning-based filter to remove unnecessary computations resulting from mismatches between the different microservices in the video analytics pipeline.

### Claim 18 (depends on 17)

The system of claim 17, wherein the graph-based filter removes repeated objects from consecutive frames within the sequence of frames by creating a graph G=(V, E), where each object forms a vertex vi and an edge e (vi, vj) between two vertices vi and vj, if and only if, these objects arc close to each other, the closeness of two objects determined by spatial closeness and temporal closeness used to compute a distance metric.

### Claim 19 (depends on 18)

The system of claim 18, wherein the spatial closeness employs an overlap ratio between bounding boxes of the objects from the consecutive frames, the overlap ratio given as:
overlap
⁢
⁢
(
v
i
,
v
j
)
=
area
⁢
⁢
(
v
i
⋂
v
j
)
min
⁡
(
area
⁢
⁢
(
v
i
)
,
area
⁢
⁢
(
v
j
)
)
.

### Claim 20 (depends on 18)

The system of claim 18, wherein the temporal closeness employs a lag between the objects from the consecutive frames, the lag given as:
lag
⁢
⁢
(
v
i
,
v
j
)
=

frame
⁢
⁢
#
⁢
v
i
-
frame
⁢
⁢
#
⁢
v
j

frameInBatch
.