# US 2017/0286776 A1 · Large margin high-order deep learning with auxiliary tasks for video-based anomaly detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0286776 A1
- Document ID: US20170286776A1
- Status: Published application
- Kind: A1
- Application number: 15/380,308
- Filed: 2016-12-15
- Published: 2017-10-05
- Earliest claimed: 2016-03-30
- Assignees: NEC Laboratories America, Inc.
- Applicants: NEC Laboratories America, Inc.
- Inventors: Renqiang Min; Dongjin Song; Eric Cosatto
- Web page: https://patentgrep.com/patent/US20170286776A1

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06F 18/24133: {Distances to prototypes}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/048: Activation functions
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06V 30/19173: {Classification techniques}
- H04N 5/21: Circuitry for suppressing or minimising disturbance, e.g. moiré or halo
- G06V 20/44: {Event detection}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A video camera is provided for video-based anomaly detection that includes at least one imaging sensor configured to capture video sequences in a workplace environment having a plurality of machines therein. The video camera further includes a processor. The processor is configured to generate one or more predictions of an impending anomaly affecting at least one item selected from the group consisting of (i) at least one of the plurality of machines and (ii) at least one operator of the at least one of the plurality of machines, using a Deep High-Order Convolutional Neural Network (DHOCNN)-based model applied to the video sequences. The DHOCNN-based model has a one-class SVM as a loss layer of the model. The processor is further configured to generate a signal for initiating an action to the at least one of the plurality of machines to mitigate expected harm to the at least one item.

## Claims

### Claim 1

A video camera for video-based anomaly detection, comprising:
at least one imaging sensor configured to capture one or more video sequences in a workplace environment having a plurality of machines therein;
a processor, configured to:
generate one or more predictions of an impending anomaly affecting at least one item selected from the group consisting of (i) at least one of the plurality of machines and (ii) at least one operator of the at least one of the plurality of machines, using a Deep High-Order Convolutional Neural Network (DHOCNN)-based model applied to the one or more video sequences, the DHOCNN-based model having a one-class Support Vector Machine (SVM) as a loss layer of the DHOCNN-based model; and
generate a signal for initiating an action to the at least one of the plurality of machines to mitigate expected harm to the at least one item.

### Claim 2 (depends on 1)

The video camera of claim 1, further comprising a network adapter configured to transmit the signal to an object selected from the group consisting of (i) the at least of the plurality of machines, and (ii) a safety device associating with the at least one of the plurality of machines.

### Claim 3 (depends on 1)

The video camera of claim 1, further comprising a hardware port for receiving a wired communication medium over which the signal is transmitted to an object selected from the group consisting of (i) the at least of the plurality of machines, and (ii) a safety device associating with the at least one of the plurality of machines.

### Claim 4 (depends on 1)

The video camera of claim 1, wherein the processor is further configured to pre-train the DHOCNN-based model using a plurality of auxiliary tasks relating to potential dangerous conditions which generate supervision labels and guide parameter learning for the DHOCNN-based model.

### Claim 5 (depends on 4)

The video camera of claim 4, wherein pre-training the DHOCNN-based model uses back-propagation with the plurality of auxiliary tasks.

### Claim 6 (depends on 1)

The video camera of claim 1, wherein the processor is further configured to fine-tune model parameters of the DHOCNN-based model using back-propagation.

### Claim 7 (depends on 1)

The video camera of claim 1, wherein the SVM is employed in a final layer of the DHOCNN-based model.

### Claim 8 (depends on 1)

The video camera of claim 1, wherein the video-based anomaly detection is supervised video-based anomaly detection.

### Claim 9 (depends on 1)

The video camera of claim 1, wherein the DHOCNN-based model uses a plurality of high-order kernels to capture high-order feature interactions relating to the input image on which the video-based anomaly detection is performed.

### Claim 10 (depends on 1)

The video camera of claim 1, wherein the processor is further configured to reduce a computational complexity of the DHOCNN-based model using a matrix factorization technique.

### Claim 11 (depends on 10)

The video camera of claim 10, wherein the matrix factorization technique involves a plurality of rank-one matrices.

### Claim 12 (depends on 11)

The video camera of claim 11, wherein the plurality of rank-one matrices are configured to explicitly model different orders of interaction among features extracted the input image.

### Claim 13 (depends on 1)

The video camera of claim 1, wherein the SVM is configured to separate data examples from an origin in a feature space and maximize a distance from a hyperplane of the feature space to the origin.

### Claim 14 (depends on 1)

The video camera of claim 1, wherein the SVM is configured to produce a binary function that captures regions in a feature space where a probability density is greatest.

### Claim 15

A video camera system for video-based anomaly detection, comprising:
a plurality of video cameras, each including:
at least one imaging sensor configured to capture one or more video sequences in a workplace environment having a plurality of machines therein;
a processor, configured to:
generate one or more predictions of an impending anomaly affecting at least one item selected from the group consisting of (i) at least one of the plurality of machines and (ii) at least one operator of the at least one of the plurality of machines, using a Deep High-Order Convolutional Neural Network (DHOCNN)-based model applied to the one or more video sequences, the DHOCNN-based model having a one-class Support Vector Machine (SVM) as a loss layer of the DHOCNN-based model; and
generate a signal for initiating an action to the at least one of the plurality of machines to mitigate expected harm to the at least one item.

### Claim 16 (depends on 15)

The video camera system of claim 15, wherein each of the plurality of video cameras further include a network adapter configured to transmit the signal to an object selected from the group consisting of (i) the at least of the plurality of machines, and (ii) a safety device associating with the at least one of the plurality of machines.

### Claim 17 (depends on 15)

The video camera system of claim 15, wherein each of the plurality of video cameras further include a hardware port for receiving a wired communication medium over which the signal is transmitted to an object selected from the group consisting of (i) the at least of the plurality of machines, and (ii) a safety device associating with the at least one of the plurality of machines.

### Claim 18 (depends on 15)

The video camera system of claim 15, wherein the processor in each of the plurality of video cameras is further configured to pre-train the DHOCNN-based model using a plurality of auxiliary tasks relating to potential dangerous conditions which generate supervision labels and guide parameter learning for the DHOCNN-based model.

### Claim 19 (depends on 18)

The video camera system of claim 18, wherein pre-training the DHOCNN-based model uses back-propagation with the plurality of auxiliary tasks.

### Claim 20 (depends on 15)

The video camera system of claim 15, wherein the processor in each of the plurality of video cameras is further configured to fine-tune model parameters of the DHOCNN-based model using back-propagation.