Quantized neural network training and inference

A system for training neural networks, comprising:

• a storage comprising a neural network model having a plurality of neurons each associated with a quantized activation function adapted to output a quantized activation value selected from a first finite set, said plurality of neurons are arranged in a plurality of layers and being connected by a plurality of connections each associated with a quantized connection weight function adapted to output a quantized connection weight value selected from a second finite set;
• at least one processor coupled to the storage for executing a code comprising:
• using a training set dataset to train said neural network model according to respective said quantized connection weight values, said training includes computing a plurality of weight gradients for backpropagation sub-processes by:
• computing a plurality of neuron gradients, each said neuron gradient is of an output of a respective said quantized activation function in one layer of said plurality of layers with respect to an input of said respective quantized activation function and is calculated such that when an absolute value of said input is smaller than a positive constant threshold value, said respective neuron gradient is set as a positive constant value and when the absolute value of said input is smaller than said positive constant threshold value said neuron gradient is set to zero, and
• updating a plurality of floating point connection weight values according to said plurality of weight gradients;
• and
• outputting a trained quantized neural network formed as an outcome of said training process.

A method for using a trained neural network, comprising:

• providing a neural network trained by a training set and having a plurality of neurons each associated with a quantized activation function adapted to output a quantized activation value selected from a first finite set, said plurality of neurons are arranged in a plurality of layers and being connected by a plurality of connections each associated with a quantized connection weight value selected from a second finite set;
• receiving new data element;
• using said neural network trained for inferencing a conclusion regarding said data element by at least one forward pass process during which only bitwise operations are held to multiply respective said quantized connection weight values and respective said quantized activation values;
• and
• outputting said conclusion.