# US 2021/0279519 A1 · Volterra Neural Network and Method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279519 A1
- Document ID: US20210279519A1
- Status: Published application
- Kind: A1
- Application number: 17/188,464
- Filed: 2021-03-01
- Published: 2021-09-09
- Earliest claimed: 2020-02-28
- Assignees: North Carolina State University
- Applicants: North Carolina State University
- Inventors: Hamid Krim; Siddharth Roheda; Sally Ghanem
- Web page: https://patentgrep.com/patent/US20210279519A1

## CPC

- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06T 7/215: Motion-based segmentation
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06N 3/047: Probabilistic or stochastic networks
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An exemplified methods and systems provides a Volterra filter network architecture that employs a cascaded implementation and a plurality of kernels, a set of which is configured to execute an nth order filter, wherein the plurality of kernels of the nth order filters are repeatedly configured in a plurality of cascading layers of interconnected kernels to form a cascading hierarchical structure that approximates a high-order filter.

## Claims

### Claim 1

A computer-implemented neural network system comprising a processor and a memory configured to execute a memory structure comprising nodes connected via a plurality of layers, the memory structure comprising:
a plurality of kernels, a set of which being configured to execute an nth order Volterra filter, wherein the plurality of kernels of the nth order Volterra filters are repeatedly configured in a plurality of cascading layers of interconnected kernels to form a cascading hierarchical structure that approximates a high-order Volterra filter.

### Claim 2 (depends on 1)

The computer-implemented neural network system of claim 1, wherein the nth order filters each comprise a 2nd or 3rd-order filter to form the cascading hierarchical structure and approximates a 4th order filter or more.

### Claim 3 (depends on 1)

The computer-implemented neural network system of claim 1, wherein the nth order filters each includes a Volterra filter that models an input-output relationship of a nonlinear system.

### Claim 4 (depends on 3)

The computer-implemented neural network system of claim 3, wherein the Volterra filter is implemented as a product operator of two 3-dimensional matrices.

### Claim 5 (depends on 1)

The computer-implemented neural network system of claim 1, wherein the cascading hierarchical structure approximates a Kth order filter from Z number of 2nd order filters according to K=22(Z−1).

### Claim 6 (depends on 1)

The computer-implemented neural network system of claim 1, wherein the plurality of cascading layers include at least a first cascade layer and a second cascade layer, wherein the output features of the first cascade layer are overlapped to provide non-linearly cascading inputs into the second cascade layer.

### Claim 7 (depends on 1)

The computer-implemented neural network system of claim 1, wherein the output features of the first cascade layer are overlapped to provide quadratic interactions between the first cascading layer and the second cascading layer.

### Claim 8 (depends on 1)

The computer-implemented neural network system of claim 1, wherein the set of the plurality of kernels each comprises a Volterra filter having an nth order filter and a filter length of at least 2.

### Claim 9 (depends on 1)

The computer-implemented neural network system of claim 1, wherein each cascading layer of the plurality of cascading layers has a same configuration.

### Claim 10 (depends on 1)

The computer-implemented neural network system of claim 1, wherein each kernel of a first cascade layer has a first configuration and each kernel of a second cascade layer has a second configuration, wherein the first configuration is different from the second configuration.

### Claim 11 (depends on 1)

The computer-implemented neural network system of claim 1, wherein the computer-implemented neural network is configured to output a plurality of predictor values each associated with a likelihood of a given recognized action being present or not present from among a set of evaluate-able recognized actions.

### Claim 12 (depends on 1)

The computer-implemented neural network system of claim 1,
wherein the plurality of kernels of the cascading hierarchical structure are used to independently assess a first data set for first modality features,
wherein the memory structure further comprises:
a second plurality of kernels, a set of which being configured to execute an mth, wherein the second plurality of kernels of the mth order filters are repeatedly configured in a plurality of cascading layer of interconnected kernels to form a second cascading hierarchical structure that approximates a high-order filter substantially greater than m, wherein the second plurality of kernels of the second cascading hierarchical structure are used to independently assess a second data set for second modality features, a fusion operator configured to fuse i) the first modality features of the first plurality of kernels in the first cascading hierarchical structure and ii) the second modality features of the second plurality of kernels in the second cascading hierarchical structure to generate output predictors.

### Claim 13 (depends on 12)

The computer-implemented neural network system of claim 12, wherein the fusion operator comprises a Volterra filter.

### Claim 14 (depends on 12)

The computer-implemented neural network system of claim 12, wherein the first and second modality features comprise spatial features and temporal features, respectively, wherein the computer-implemented neural network further comprises a classifier to output indication of presence or non-presence of a recognized action in a set of video frames or an image sequence having representations of one or more action therein based on the spatial features and temporal features.

### Claim 15 (depends on 1)

The computer-implemented neural network system of claim 1, wherein the first data set comprise a set of video frames.

### Claim 16 (depends on 14)

The computer-implemented neural network system of claim 14, wherein the set of video frames comprises red, green, and blue (RGB) data values.

### Claim 17 (depends on 12)

The computer-implemented neural network system of claim 12, wherein the second modality features are associated with an optical flow analysis.

### Claim 18 (depends on 12)

The computer-implemented neural network system of claim 12, wherein the fusion operator evaluate non-linear interactions between the first modality features and the second modality features.

### Claim 19 (depends on 1)

The computer-implemented neural network system of claim 1, wherein the plurality of kernels and cascading hierarchical structure are configured as a neural network auto-encoder.

### Claim 20 (depends on 1)

The computer-implemented neural network system of claim 1, wherein the plurality of kernels and cascading hierarchical structure are configured as a generative adversarial network.

### Claim 21

A method comprising:
training, via a computing system, a Volterra neural network comprising plurality of kernels each configured to execute an nth order filter, wherein the plurality of kernels of the nth order filters are repeatedly configured in a plurality of cascading layers of interconnected kernels to form a cascading hierarchical structure that approximates a high-order filter substantially greater than n; and
using the trained neural network in a video or image processing application.

### Claim 22

A non-transitory computer readable medium comprising instructions, wherein execution of the instructions, by a processor, cause the processor to:
train a Volterra neural network comprising plurality of kernels each configured to execute an nth order filter, wherein the plurality of kernels of the nth order filters are repeatedly configured in a plurality of cascading layers of interconnected kernels to form a cascading hierarchical structure that approximates a high-order filter substantially greater than n; and and
use the trained neural network in a video or image processing application.