# US 2017/0289409 A1 · Large margin high-order deep learning with auxiliary tasks for video-based anomaly detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0289409 A1
- Document ID: US20170289409A1
- Status: Published application
- Kind: A1
- Application number: 15/380,014
- Filed: 2016-12-15
- Published: 2017-10-05
- Earliest claimed: 2016-03-30
- Assignees: NEC Laboratories America, Inc.
- Applicants: NEC Laboratories America, Inc.
- Inventors: Renqiang Min; Dongjin Song; Eric Cosatto
- Web page: https://patentgrep.com/patent/US20170289409A1

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06F 18/24133: {Distances to prototypes}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/048: Activation functions
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 30/19173: {Classification techniques}
- H04N 5/21: Circuitry for suppressing or minimising disturbance, e.g. moiré or halo
- G06V 20/44: {Event detection}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A computer-implemented method and system are provided for video-based anomaly detection. The method includes forming, by a processor, a Deep High-Order Convolutional Neural Network (DHOCNN)-based model having a one-class Support Vector Machine (SVM) as a loss layer of the DHOCNN-based model. An objective of the SVM is configured to perform the video-based anomaly detection. The method further includes generating, by the processor, one or more predictions of an impending anomaly based on the high-order deep learning based model applied to an input image. The method also includes initiating, by the processor, an action to a hardware device to mitigate expected harm to at least one item selected from the group consisting of the hardware device, another hardware device related to the hardware device, and a person related to the hardware device.

## Claims

### Claim 1

A computer-implemented method for video-based anomaly detection, comprising:
forming, by a processor, a Deep High-Order Convolutional Neural Network (DHOCNN)-based model having a one-class Support Vector Machine (SVM) as a loss layer of the DHOCNN-based model, an objective of the SVM being configured to perform the video-based anomaly detection;
generating, by the processor, one or more predictions of an impending anomaly based on the high-order deep learning based model applied to an input image; and
initiating, by the processor, an action to a hardware device to mitigate expected harm to at least one item selected from the group consisting of the hardware device, another hardware device related to the hardware device, and a person related to the hardware device.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising pre-training the DHOCNN-based model using a plurality of auxiliary tasks relating to potential dangerous conditions which generate supervision labels and guide parameter learning for the DHOCNN-based model.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein pre-training the DHOCNN-based model uses back-propagation with the plurality of auxiliary tasks.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, further comprising fine-tuning model parameters of the DHOCNN-based model using back-propagation.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the SVM is employed in a final layer of the DHOCNN-based model.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the video-based anomaly detection is supervised video-based anomaly detection.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the DHOCNN-based model uses a plurality of high-order kernels to capture high-order feature interactions relating to the input image on which the video-based anomaly detection is performed.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, further comprising reducing a computational complexity of the DHOCNN-based model using a matrix factorization technique.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein the matrix factorization technique involves a plurality of rank-one matrices.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein the plurality of rank-one matrices are configured to explicitly model different orders of interaction among features extracted the input image.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, wherein the SVM is configured to separate data examples from an origin in a feature space and maximize a distance from a hyperplane of the feature space to the origin.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, wherein the SVM is configured to produce a binary function that captures regions in a feature space where a probability density is greatest.

### Claim 13

A computer program product for video-based anomaly detection, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
forming, by a processor, a Deep High-Order Convolutional Neural Network (DHOCNN)-based model having a one-class Support Vector Machine (SVM) as a loss layer of the DHOCNN-based model, an objective of the SVM being configured to perform the video-based anomaly detection;
generating, by the processor, one or more predictions of an impending anomaly based on the high-order deep learning based model applied to an input image; and
initiating, by the processor, an action to a hardware device to mitigate expected harm to at least one item selected from the group consisting of the hardware device, another hardware device related to the hardware device, and a person related to the hardware device.

### Claim 14 (depends on 13)

The computer program product of claim 13, wherein the method further comprises pre-training the DHOCNN-based model using a plurality of auxiliary tasks relating to potential dangerous conditions which generate supervision labels and guide parameter learning for the DHOCNN-based model.

### Claim 15 (depends on 13)

The computer program product of claim 13, wherein the method further comprises fine-tuning model parameters of the DHOCNN-based model using back-propagation.

### Claim 16 (depends on 13)

The computer program product of claim 13, wherein the SVM is employed in a final layer of the DHOCNN-based model.

### Claim 17 (depends on 13)

The computer program product of claim 13, wherein the method further comprises reducing a computational complexity of the DHOCNN-based model using a matrix factorization technique that involves a plurality of rank-one matrices, and wherein the plurality of rank-one matrices are configured to explicitly model different orders of interaction among features extracted the input image.

### Claim 18 (depends on 13)

The computer program product of claim 13, wherein the SVM is configured to separate data examples from an origin in a feature space and maximize a distance from a hyperplane of the feature space to the origin.

### Claim 19 (depends on 13)

The computer program product of claim 13, wherein the SVM is configured to produce a binary function that captures regions in a feature space where a probability density is greatest.

### Claim 20

A computer processing system for performing video-based anomaly detection, comprising:
a processor, configured to:
form a Deep High-Order Convolutional Neural Network (DHOCNN)-based model having a one-class Support Vector Machine (SVM) as a loss layer of the DHOCNN-based model, an objective of the SVM being configured to perform the video-based anomaly detection;
generate one or more predictions of an impending anomaly based on the high-order deep learning based model applied to an input image; and
initiate an action to a hardware device to mitigate expected harm to at least one item selected from the group consisting of the hardware device, another hardware device related to the hardware device, and a person related to the hardware device.
