# US 2023/0267589 A1 · Method and apparatus for multi-frame prediction error-based video anomaly detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0267589 A1
- Document ID: US20230267589A1
- Status: Published application
- Kind: A1
- Application number: 18/086,795
- Filed: 2022-12-22
- Published: 2023-08-24
- Earliest claimed: 2022-02-23
- Assignees: Industry-Academia Cooperation Group of Sejong University
- Applicants: Industry-Academia Cooperation Group Of Sejong University
- Inventors: Young-Gab KIM; Yujun KIM
- Web page: https://patentgrep.com/patent/US20230267589A1

## CPC

- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06T 7/254: involving subtraction of images
- G08B 13/196: using television cameras
- G08B 31/00: Predictive alarm systems characterised by extrapolation or other computation using updated historic data
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30168: Image quality inspection
- G06T 2207/30232: Surveillance
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS

## Abstract

Disclosed is a method and apparatus for multi-frame prediction error-based video anomaly detection. The disclosed apparatus for multi-frame prediction error-based video anomaly detection includes a multi-frame predictor configured to predict a current frame and adjacent frames using a prediction model; a multi-frame prediction error obtainer configured to obtain a multi-frame prediction error by obtaining a prediction error of the current frame and prediction errors of the adjacent frames that each prediction error is a difference between a predicted frame and a real frame and by adding the prediction error of the current frame and the prediction errors of the adjacent frames; and an anomaly score evaluator configured to evaluate an anomaly score based on the obtained multi-frame prediction error of the current frame.

## Claims

### Claim 1

A video anomaly detection apparatus comprising:
a multi-frame predictor configured to predict a current frame and adjacent frames using a prediction model;
a multi-frame prediction error obtainer configured to obtain a multi-frame prediction error by obtaining a prediction error of the current frame and prediction errors of the adjacent frames that each prediction error is a difference between a predicted frame and a real frame and by adding the prediction error of the current frame and the prediction errors of the adjacent frames; and
an anomaly score evaluator configured to evaluate an anomaly score based on the obtained multi-frame prediction error of the current frame.

### Claim 2 (depends on 1)

The video anomaly detection apparatus of claim 1, wherein, to predict the current frame and the adjacent frames, the multi-frame predictor is configured to predict the current frame and the adjacent frames by receiving and concatenating a previous frame and a next frame as color through two sub-networks for each of the current frame and the adjacent frames.

### Claim 3 (depends on 1)

The video anomaly detection apparatus of claim 1, wherein the multi-frame prediction error obtainer is configured to obtain prediction errors for the current frame and the adjacent frames by squaring a pixel difference between the predicted frame and the real frame for each frame and to obtain a multi-frame prediction error that is a sum of the prediction error of the current frame and the prediction errors of the adjacent frames to reduce an incorrect prediction error gap.

### Claim 4 (depends on 3)

The video anomaly detection apparatus of claim 3, wherein the multi-frame prediction error obtainer is configured to obtain a multi-frame prediction error for the current frame by multiplying the prediction error by a weight for reducing influence of a prediction error of a corresponding adjacent frame on anomaly detection of the current frame as a distance between the adjacent frame and the current frame increases and then summing each weight-multiplied prediction error.

### Claim 5 (depends on 4)

The video anomaly detection apparatus of claim 4, wherein an equation for calculating the multi-frame prediction error MPEi,jt is represented as follows:
MPE
i
,
j
t
=
∑
k
=
t
-
n
t
+
n
w
❘
"\[LeftBracketingBar]"
k
-
t
❘
"\[RightBracketingBar]"
·
PE
i
,
j
t
where t denotes a time, w denotes a weight, and PEi,jt denotes a prediction error.

### Claim 6 (depends on 1)

The video anomaly detection apparatus of claim 1, wherein the anomaly score evaluator is configured to obtain the anomaly score for the current frame using a cascade sliding window method for the obtained multi-frame prediction error.

### Claim 7 (depends on 6)

The video anomaly detection apparatus of claim 6, wherein the cascade sliding window method moves a single window in the multi-frame prediction error and calculates a mean for a patch of the multi-frame prediction error corresponding to the window every time the window moves, and after moving the window evenly in both sides of the entire multi-frame prediction error, calculates the anomaly score by selecting and averaging a portion of the calculated mean.

### Claim 8

A video anomaly detection method comprising:
predicting, by a multi-frame predictor, a current frame and adjacent frames using a prediction model;
obtaining, by a multi-frame prediction error obtainer, a multi-frame prediction error by obtaining a prediction error of the current frame and prediction errors of the adjacent frames that each prediction error is a difference between a predicted frame and a real frame and by adding the prediction error of the current frame and the prediction errors of the adjacent frames; and
evaluating, by an anomaly score evaluator, an anomaly score based on the obtained multi-frame prediction error of the current frame.

### Claim 9 (depends on 8)

The video anomaly detection method of claim 8, wherein the predicting comprises predicting the current frame and the adjacent frames by receiving and concatenating a previous frame and a next frame as color through two sub-networks for each of the current frame and the adjacent frames.

### Claim 10 (depends on 8)

The video anomaly detection method of claim 8, wherein the obtaining comprises obtaining a prediction error for the current frame and the adjacent frames by squaring a pixel difference between the predicted current frame and the real frame of the adjacent frames and obtaining a multi-frame prediction error that is a sum of the prediction error of the current frame and the prediction errors of the adjacent frames to reduce an incorrect prediction error gap.

### Claim 11 (depends on 10)

The video anomaly detection method of claim 10, wherein the obtaining comprises obtaining a multi-frame prediction error for the current frame by multiplying the prediction error by a weight for reducing influence of a prediction error of a corresponding adjacent frame on anomaly detection of the current frame as a distance between the adjacent frame and the current frame increases and then summing each weight-multiplied prediction error.

### Claim 12 (depends on 11)

The video anomaly detection method of claim 11, wherein an equation for calculating the multi-frame prediction error MPEi,jt is represented as follows:
MPE
i
,
j
t
=
∑
k
=
t
-
n
t
+
n
w
❘
"\[LeftBracketingBar]"
k
-
t
❘
"\[RightBracketingBar]"
·
PE
i
,
j
t
where t denotes a time, w denotes a weight, and PEi,jt denotes a prediction error.

### Claim 13 (depends on 8)

The video anomaly detection method of claim 8, wherein the evaluating comprises obtaining the anomaly score for the current frame using a cascade sliding window method for the obtained multi-frame prediction error.

### Claim 14 (depends on 13)

The video anomaly detection method of claim 13, wherein the cascade sliding window method moves a single window in the multi-frame prediction error and calculates a mean for a patch of the multi-frame prediction error corresponding to the window every time the window moves, and after moving the window evenly in both sides of the entire multi-frame prediction error, calculates the anomaly score by selecting and averaging a portion of the calculated mean.
