# US 12,602,573 B2 · Neural network robustness via binary activation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,602,573 B2
- Document ID: US12602573B2
- Status: Granted
- Kind: B2
- Application number: 17/314,751
- Filed: 2021-05-07
- Published: 2021-11-11
- Granted: 2026-04-14
- Earliest claimed: 2018-09-28
- Assignees: National Technology & Engineering Solutions of Sandia, LLC
- Applicants: National Technology & Engineering Solutions of Sandia, LLC
- Inventors: William Mark Severa; Craig Michael Vineyard; Ryan Anthony Dellana; Abrar Anwar
- Web page: https://patentgrep.com/patent/US12602573B2

## CPC

- G06N 3/063: using electronic means
- G06F 18/2113: by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/048: Activation functions
- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/086: using evolutionary algorithms, e.g. genetic algorithms or genetic programming
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method of increasing neural network robustness. The method comprises defining an artificial neural network comprising a number of bounded ramp activation functions. The network is trained iteratively in a layer-by-layer fashion. Each iteration increases the slope of the activation functions toward a discrete threshold activation and stops when the activation functions converge to the threshold activation and the network exhibits spiking behavior. Alternatively, weight agnostic neural networks are created, wherein nodes in the networks comprise fixed shared weights. A subset of networks is identified that comprise activation functions compatible with neuromorphic hardware and are tested with a specified number of shared weight values. A score is generated for each combination of network and weight value according to performance and mapping to neuromorphic hardware, and the networks are ranked. The networks are then combined according to ranking to create a new network that exhibits spiking behavior.

## Claims

### Claim 1

A method of increasing a robustness of a neural network, the method comprising providing a drop-in mechanism for tailoring a deep neural network into a spiking hardware platform or other binary threshold activation platform by using a number of processors to perform the steps of:
defining an artificial neural network, wherein the artificial neural network is weight agnostic and comprises a number of bounded ramp activation functions;
iteratively training the artificial neural network into a spiking network in a layer-by-layer fashion, wherein each iteration of training increases g slope of the bounded ramp activation functions toward a discrete threshold activation, wherein iterations of training incorporate constraints selectively targeting discrete communication between nodes and thereby pushing the artificial neural network towards discrete spike activations shifting a gradient of bounded rectified linear unit activations incrementally toward a discrete perceptron-like step followed by fine-tuning of the artificial neural network to account for any losses while pushing towards the discrete spike activations;
stopping training of the artificial neural network when the number of bounded ramp activation functions converge to the discrete threshold activation and the artificial neural network exhibits spiking behavior;
implementing the spiking network in neuromorphic hardware; and
testing the spiking network for transient upsets or progressive degradation by injecting noise into the neural network by wrapping a memory using a noise generator.

### Claim 2 (depends on 1)

The method of claim 1, further comprising feeding noise into the neural network during training and, responsive to the noise, adjusting hyperparameters of the artificial neural network.

### Claim 3 (depends on 2)

The method of claim 2, further comprising adjusting neural network hyperparameters in response to the noise.

### Claim 4 (depends on 2)

The method of claim 2, further comprising adjusting a training dataset in response to the noise.

### Claim 5 (depends on 2)

The method of claim 2, further comprising modifying an output of an artificial neuron responding to the noise by adjusting node weights automatically.

### Claim 6 (depends on 1)

The method of claim 1, further comprising feeding noise into the neural network after training.

### Claim 7

A computer program product configured to increase a robustness of a neural network that provides a drop-in mechanism configured to tailor a deep neural network into a spiking hardware platform or other binary threshold activation platform, wherein the computer program product comprises:
a computer-readable storage medium having that comprises program instructions embodied thereon and configured to:
define an artificial neural network, wherein the artificial neural network is weight agnostic and comprises a number of bounded ramp activation functions;
iteratively train the artificial neural network into a spiking network in a layer-by-layer fashion, wherein each iteration of training increases a slope of the bounded ramp activation functions toward a discrete threshold activation, wherein iterations of training incorporate constraints that selectively target discrete communication between nodes and thereby push the artificial neural network towards discrete spike activations that shift a gradient of bounded rectified linear unit activations incrementally toward a discrete perceptron-like step;
fine-tune the artificial neural network to account for any losses incorporated into the push towards the discrete spike activations;
stop a training of the artificial neural network when the number of bounded ramp activation functions converge to the discrete threshold activation and the artificial neural network exhibits spiking behavior;
implement the spiking network in neuromorphic hardware; and
test the spiking network for transient upsets or progressive degradation with an injection of noise into the neural network with a memory wrapped by a noise generator.

### Claim 8 (depends on 7)

The computer program product of claim 7, further comprising instructions for feeding noise into the neural network during training.

### Claim 9 (depends on 7)

The computer program product of claim 7, further comprising instructions for feeding noise into the neural network after training.

### Claim 10

A system configured to increase a robustness of a neural network based upon a drop-in mechanism configured to tailor a deep neural network into a spiking hardware platform or other binary threshold activation platform robustness, wherein the system comprises:
a storage device configured to store program instructions; and
one or more processors operably connected to the storage device and configured to execute the program instructions configured to cause the system to:
define an artificial neural network, wherein the artificial neural network is weight agnostic and comprises a number of bounded ramp activation functions;
iteratively train the artificial neural network into a spiking network in a layer-by-layer fashion, wherein each iteration of training increases a slope of the bounded ramp activation functions toward a discrete threshold activation, wherein iterations of training incorporate constraints that selectively target discrete communication between nodes and thereby push the artificial neural network towards discrete spike activations that shift a gradient of bounded rectified linear unit activations incrementally toward a discrete perceptron-like step;
fine-tune the artificial neural network to account for any losses incorporated into the push towards the discrete spike activations;
stop a training of the artificial neural network when the number of bounded ramp activation functions converge to the discrete threshold activation and the artificial neural network exhibits spiking behavior;
implement the spiking network in neuromorphic hardware; and
test the spiking network for transient upsets or progressive degradation with an injection of noise into the neural network with a memory wrapped by a noise generator.

### Claim 11 (depends on 10)

The system of claim 10, wherein noise is fed into the neural network during training.

### Claim 12 (depends on 10)

The system of claim 10, wherein noise is fed into the neural network after training.

### Claim 13

A method of increasing a robustness of a neural network, the method comprising using a number of processors providing a drop-in mechanism for tailoring a neural network into a spiking hardware platform or other binary threshold activation platform by performing the steps of:
using iterations of training that incorporate constraints selectively targeting discrete communication between nodes and thereby pushing the neural network towards discrete spike activations shifting a gradient of bounded rectified linear unit activations incrementally toward a discrete perceptron-like step followed by fine-tuning of the neural network to account for any losses while pushing towards the discrete spike activations for creating a number of minimal weight agnostic neural networks, wherein nodes in the number of minimal weight agnostic neural networks comprise fixed shared weights;
identifying a subset of the number of minimal weight agnostic neural networks that comprise activation functions compatible with neuromorphic hardware;
implementing a spiking network in the neuromorphic hardware;
testing each neural network in the subset with a specified number of shared weight values for transient upsets or progressive degradation by injecting noise into the neural network by wrapping a memory using a noise generator;
generating a score for each combination of a neural network in the subset and a shared weight value, wherein the score is based on performance and complexity in mapping to neuromorphic hardware;
ranking all neural networks in the subset according to their respective scores; and
combining neural networks from the subset according to ranking to create a new network that exhibits spiking behavior.

### Claim 14 (depends on 13)

The method of claim 13, wherein the activation functions in the subset of neural networks comprise linear functions and threshold activation functions.

### Claim 15 (depends on 13)

The method of claim 13, wherein the subset of neural networks are tested according to parameters comprising at least one of:
accuracy;
speed;
energy consumption; or
size and complexity.

### Claim 16

A computer program product configured to increase a robustness of a neural network, based upon a drop-in mechanism configured to tailor a deep neural network into a spiking hardware platform or other binary threshold activation platform and the computer program product comprising a computer-readable storage medium that comprises program instructions configured to perform the steps of:
selectively targeting, based on iterations of training incorporating constraints, discrete communication between nodes and thereby push the neural network towards discrete spike activations that shift a gradient of bounded rectified linear unit activations incrementally toward a discrete perceptron-like step followed by a fine-tuning of the neural network to account for any losses while pushing towards the discrete spike activations for creating a number of minimal weight agnostic neural networks, wherein nodes in the number of minimal weight agnostic neural networks comprise fixed shared weights;
identifying a subset of the number of minimal weight agnostic neural networks that comprise activation functions compatible with neuromorphic hardware;
implementing a spiking network in the neuromorphic hardware;
testing each neural network in the subset with a specified number of shared weight values for transient upsets or progressive degradation by injecting noise into the neural network by wrapping a memory using a noise generator;
generating a score for each combination of a neural network in the subset and a shared weight value, wherein the score is based on performance and complexity in mapping to neuromorphic hardware;
ranking all neural networks in the subset according to their respective scores; and combining neural networks from the subset according to ranking to create a new network that exhibits spiking behavior.

### Claim 17 (depends on 16)

The computer program product of claim 16, wherein the activation functions in the subset of neural networks comprise linear functions and threshold activation functions.

### Claim 18 (depends on 16)

The computer program product of claim 16, wherein the subset of neural networks are tested according to parameters comprising at least one of:
accuracy;
speed;
energy consumption; or
size and complexity.

### Claim 19

A system configured to increase a robustness of a neural network based upon a drop-in mechanism configured to tailor a deep neural network into a spiking hardware platform or other binary threshold activation platform, the system comprising:
a storage device configured to store program instructions; and
one or more processors operably connected to the storage device and configured to execute the program instructions to cause the system to:
create a number of minimal weight agnostic neural networks, wherein nodes in the number of minimal weight agnostic neural networks comprise fixed shared weights by using iterations of training that incorporate constraints that electively target discrete communication between nodes and thereby push the number of minimal weight agnostic neural networks towards discrete spike activations that shift a gradient of bounded rectified linear unit activations incrementally toward a discrete perceptron-like step followed by a fine-tuning of the number of minimal weight agnostic neural network that accounts for any losses while pushing towards the discrete spike activations;
identify a subset of the number of minimal weight agnostic neural networks that comprise activation functions compatible with neuromorphic hardware;
implement a spiking network in neuromorphic hardware;
test each neural network in the subset with a specified number of shared weight values for transient upsets or progressive degradation with an injection of noise into the neural network with a memory wrapped by a noise generator;
generate a score for each combination of a neural network in the subset and a shared weight value, wherein the score is based on performance and complexity in mapping to neuromorphic hardware;
rank all neural networks in the subset according to their respective scores; and
combine neural networks from the subset according to ranking to create a new network that exhibits spiking behavior.

### Claim 20 (depends on 19)

The system of claim 19, wherein the activation functions in the subset of neural networks comprise linear functions and threshold activation functions.

### Claim 21 (depends on 19)

The system if claim 19, wherein the subset of neural networks are tested according to parameters comprising at least one of:
accuracy;
speed;
energy consumption; or
size and complexity.