Neural network robustness via binary activation

A computer program product configured to increase a robustness of a neural network that provides a drop-in mechanism configured to tailor a deep neural network into a spiking hardware platform or other binary threshold activation platform, wherein the computer program product comprises:

• a computer-readable storage medium having that comprises program instructions embodied thereon and configured to:
• define an artificial neural network, wherein the artificial neural network is weight agnostic and comprises a number of bounded ramp activation functions;
• iteratively train the artificial neural network into a spiking network in a layer-by-layer fashion, wherein each iteration of training increases a slope of the bounded ramp activation functions toward a discrete threshold activation, wherein iterations of training incorporate constraints that selectively target discrete communication between nodes and thereby push the artificial neural network towards discrete spike activations that shift a gradient of bounded rectified linear unit activations incrementally toward a discrete perceptron-like step;
• fine-tune the artificial neural network to account for any losses incorporated into the push towards the discrete spike activations;
• stop a training of the artificial neural network when the number of bounded ramp activation functions converge to the discrete threshold activation and the artificial neural network exhibits spiking behavior;
• implement the spiking network in neuromorphic hardware;
• and
• test the spiking network for transient upsets or progressive degradation with an injection of noise into the neural network with a memory wrapped by a noise generator.

A system configured to increase a robustness of a neural network based upon a drop-in mechanism configured to tailor a deep neural network into a spiking hardware platform or other binary threshold activation platform robustness, wherein the system comprises:

• a storage device configured to store program instructions;
• and
• one or more processors operably connected to the storage device and configured to execute the program instructions configured to cause the system to:
• define an artificial neural network, wherein the artificial neural network is weight agnostic and comprises a number of bounded ramp activation functions;
• iteratively train the artificial neural network into a spiking network in a layer-by-layer fashion, wherein each iteration of training increases a slope of the bounded ramp activation functions toward a discrete threshold activation, wherein iterations of training incorporate constraints that selectively target discrete communication between nodes and thereby push the artificial neural network towards discrete spike activations that shift a gradient of bounded rectified linear unit activations incrementally toward a discrete perceptron-like step;
• fine-tune the artificial neural network to account for any losses incorporated into the push towards the discrete spike activations;
• stop a training of the artificial neural network when the number of bounded ramp activation functions converge to the discrete threshold activation and the artificial neural network exhibits spiking behavior;
• implement the spiking network in neuromorphic hardware;
• and
• test the spiking network for transient upsets or progressive degradation with an injection of noise into the neural network with a memory wrapped by a noise generator.

A method of increasing a robustness of a neural network, the method comprising using a number of processors providing a drop-in mechanism for tailoring a neural network into a spiking hardware platform or other binary threshold activation platform by performing the steps of:

• using iterations of training that incorporate constraints selectively targeting discrete communication between nodes and thereby pushing the neural network towards discrete spike activations shifting a gradient of bounded rectified linear unit activations incrementally toward a discrete perceptron-like step followed by fine-tuning of the neural network to account for any losses while pushing towards the discrete spike activations for creating a number of minimal weight agnostic neural networks, wherein nodes in the number of minimal weight agnostic neural networks comprise fixed shared weights;
• identifying a subset of the number of minimal weight agnostic neural networks that comprise activation functions compatible with neuromorphic hardware;
• implementing a spiking network in the neuromorphic hardware;
• testing each neural network in the subset with a specified number of shared weight values for transient upsets or progressive degradation by injecting noise into the neural network by wrapping a memory using a noise generator;
• generating a score for each combination of a neural network in the subset and a shared weight value, wherein the score is based on performance and complexity in mapping to neuromorphic hardware;
• ranking all neural networks in the subset according to their respective scores;
• and
• combining neural networks from the subset according to ranking to create a new network that exhibits spiking behavior.

A computer program product configured to increase a robustness of a neural network, based upon a drop-in mechanism configured to tailor a deep neural network into a spiking hardware platform or other binary threshold activation platform and the computer program product comprising a computer-readable storage medium that comprises program instructions configured to perform the steps of:

• selectively targeting, based on iterations of training incorporating constraints, discrete communication between nodes and thereby push the neural network towards discrete spike activations that shift a gradient of bounded rectified linear unit activations incrementally toward a discrete perceptron-like step followed by a fine-tuning of the neural network to account for any losses while pushing towards the discrete spike activations for creating a number of minimal weight agnostic neural networks, wherein nodes in the number of minimal weight agnostic neural networks comprise fixed shared weights;
• identifying a subset of the number of minimal weight agnostic neural networks that comprise activation functions compatible with neuromorphic hardware;
• implementing a spiking network in the neuromorphic hardware;
• testing each neural network in the subset with a specified number of shared weight values for transient upsets or progressive degradation by injecting noise into the neural network by wrapping a memory using a noise generator;
• generating a score for each combination of a neural network in the subset and a shared weight value, wherein the score is based on performance and complexity in mapping to neuromorphic hardware;
• ranking all neural networks in the subset according to their respective scores;
• and combining neural networks from the subset according to ranking to create a new network that exhibits spiking behavior.

A system configured to increase a robustness of a neural network based upon a drop-in mechanism configured to tailor a deep neural network into a spiking hardware platform or other binary threshold activation platform, the system comprising:

• a storage device configured to store program instructions;
• and
• one or more processors operably connected to the storage device and configured to execute the program instructions to cause the system to:
• create a number of minimal weight agnostic neural networks, wherein nodes in the number of minimal weight agnostic neural networks comprise fixed shared weights by using iterations of training that incorporate constraints that electively target discrete communication between nodes and thereby push the number of minimal weight agnostic neural networks towards discrete spike activations that shift a gradient of bounded rectified linear unit activations incrementally toward a discrete perceptron-like step followed by a fine-tuning of the number of minimal weight agnostic neural network that accounts for any losses while pushing towards the discrete spike activations;
• identify a subset of the number of minimal weight agnostic neural networks that comprise activation functions compatible with neuromorphic hardware;
• implement a spiking network in neuromorphic hardware;
• test each neural network in the subset with a specified number of shared weight values for transient upsets or progressive degradation with an injection of noise into the neural network with a memory wrapped by a noise generator;
• generate a score for each combination of a neural network in the subset and a shared weight value, wherein the score is based on performance and complexity in mapping to neuromorphic hardware;
• rank all neural networks in the subset according to their respective scores;
• and
• combine neural networks from the subset according to ranking to create a new network that exhibits spiking behavior.