# US 2019/0244076 A1 · Neural Network with Embedded Filter Layer

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0244076 A1
- Document ID: US20190244076A1
- Status: Published application
- Kind: A1
- Application number: 15/888,812
- Filed: 2018-02-05
- Published: 2019-08-08
- Earliest claimed: 2018-02-05
- Assignees: Nuance Communications, Inc.
- Applicants: Nuance Communications, Inc.
- Inventors: Nasr Madi; Neil D. Barrett
- Web page: https://patentgrep.com/patent/US20190244076A1

## CPC

- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

A neural network embodiment comprises an input layer, an output layer and a filter layer. Each unit of the filter layer is configured to receive a filter layer input from a single preceding unit via a respective filter layer input connection. Each filter layer input connection is coupled to a different single preceding unit. The filter layer is configured to incentivize the neural network to learn to produce a target output from the output layer for a given input to the input layer while simultaneously learning weights for each filter layer input connection. The weights learned cause the filter layer to reduce a number of filter layer units that pass respective filter layer inputs as non-zero values. When applied as an initial internal layer between an input layer and an output layer, the filter layer incentivizes the neural network to learn which neural network input features to discard to produce the target output.

## Claims

### Claim 1

A neural network, comprising:
an input layer;
an output layer; and
a filter layer, each unit thereof configured to receive a filter layer input from a single preceding unit via a respective filter layer input connection, each filter layer input connection coupled to a different single preceding unit, the filter layer configured to incentivize the neural network to learn to produce a target output from the output layer for a given input to the input layer while simultaneously learning weights for each filter layer input connection, the weights learned causing the filter layer to reduce a number of filter layer units that pass respective filter layer inputs as non-zero values.

### Claim 2 (depends on 1)

The neural network of claim 1, wherein at least one of the weights learned has a negative value; a mean of the weights learned is a given average value; and wherein each unit of the filter layer is configured to apply a rectified linear unit (ReLU) activation function to force all outputs from the unit to zero in an event a weight learned for its respective filter layer input connection is negative.

### Claim 3 (depends on 1)

The neural network of claim 1, wherein the filter layer is further configured to incentivize the neural network to minimize a regularized loss function that is expressed on the filter layer that combines an underlying loss function with a penalty function imposed on the weights learned.

### Claim 4 (depends on 3)

The neural network of claim 3, wherein the regularized loss function is deemed minimized in an event (i) a difference between an actual output from the output layer and the target output is less than or equal to a given acceptable difference and (ii) the weights learned have an average value that matches a given value causing the penalty function to achieve its minimum.

### Claim 5 (depends on 1)

The neural network of claim 1, wherein the filter layer is a feature selection layer that is an initial hidden layer of the neural network and each filter layer input is an input feature to the neural network

### Claim 6 (depends on 1)

The neural network of claim 1, wherein the filter layer is a last hidden layer of the neural network.

### Claim 7 (depends on 1)

The neural network of claim 1, wherein the filter layer has input connections and output connections to respective units of internal layers that are neither input layers nor output layers.

### Claim 8 (depends on 1)

The neural network of claim 1, wherein each single preceding unit is located in a common preceding layer that precedes the filter layer in the neural network.

### Claim 9 (depends on 1)

The neural network of claim 1, wherein at least two units of the filter layer are configured to receive respective filter layer inputs from corresponding single preceding units located in different preceding layers that precede the filter layer in the neural network.

### Claim 10 (depends on 1)

The neural network of claim 1, wherein the filter layer is a first filter layer and the neural network further comprises at least one other filter layer.

### Claim 11 (depends on 1)

The neural network of claim 1, wherein the neural network is a densely connected neural network that includes the filter layer integrated therein.

### Claim 12

A method for filtering in a neural network, the method comprising:
incentivizing the neural network, via a filter layer integrated within the neural network, to learn to produce a target output from an output layer for a given input to an input layer while simultaneously learning weights for each filter layer input connection, each unit of the filter layer configured to receive a filter layer input from a single preceding unit via a respective filter layer input connection, each filter layer input connection coupled to a different single preceding unit; and
learning the weights for each filter layer input connection to the filter layer, the weights learned causing the filter layer to reduce a number of filter layer units of the filter layer that pass respective filter layer inputs as non-zero values.

### Claim 13 (depends on 12)

The method of claim 12, further comprising applying a rectified linear unit (ReLU) activation function by each unit to force all outputs from the unit to zero in an event a weight learned for its respective filter layer input connection is negative, wherein at least one of the weights learned has a negative value and wherein a mean of the weights learned is a given average value.

### Claim 14 (depends on 12)

The method of claim 12, further comprising expressing a regularized loss function on the filter layer that combines an underlying loss function with a penalty function imposed on the weights learned, wherein incentivizing the neural network includes incentivizing the neural network to minimize the regularized loss function.

### Claim 15 (depends on 14)

The method of claim 14, further comprising deeming the regularized loss function to be minimized in an event (i) a difference between an actual output from the output layer and the target output is less than or equal to a given acceptable difference and (ii) the weights learned have an average value that matches a given value causing the penalty function to achieve its minimum.

### Claim 16 (depends on 12)

The method of claim 12, wherein the filter layer is a feature selection layer that is an initial hidden layer of the neural network and each filter layer input is an input feature to the neural network.

### Claim 17 (depends on 12)

The method of claim 12, wherein the filter layer is a last hidden layer of the neural network or wherein the filter layer has input connections and output connections to respective units of internal layers that are neither input layers nor output layers.

### Claim 18 (depends on 12)

The method of claim 12, further comprising including each single preceding unit in a common preceding layer that precedes the filter layer in the neural network or configuring at least two units of the filter layer to receive respective filter layer inputs from corresponding single preceding units included in different preceding layers that precede the filter layer in the neural network.

### Claim 19 (depends on 12)

The method of claim 12, wherein the filter layer is a first filter layer and the method further comprises including at least one other filter layer in the neural network.

### Claim 20 (depends on 12)

The method of claim 12, wherein the neural network is a densely connected neural network that includes the filter layer integrated therein.
