Activation function computation for neural networks

A system for implementing a machine learning function, the system comprising:

• at least one rectifier linear unit (ReLU);
• and
• at least one controller coupled with the at least one ReLU and is configured to perform a method for computing an activation function, the method comprising:
• initializing a plurality of weights in a weight vector;
• receiving an input vector with a plurality of input values for computing the dot product of the input vector with the weight vector;
• predicting that an output value of the activation function will be negative based on computing an intermediate value for computing the dot product, and based on a magnitude of the intermediate value exceeding a precomputed threshold value;
• and
• in response to the prediction, terminating the computation of the dot product, and outputting a 0 as the result of the activation function.

A rectifier linear unit (ReLU) comprising:

• a storage medium;
• a comparator;
• a plurality of multipliers;
• and
• an adder tree;
• wherein, the ReLU is configured to perform a method for computing an activation function, the method comprising:
• receiving a plurality of weights values in to setup a weight vector;
• receiving an input vector with a plurality of input values for computing a dot product of the input vector and the weight vector;
• predicting that an output value of the activation function will be negative based on computing an intermediate value for computing the dot product, and based on a magnitude of the intermediate value exceeding a precomputed threshold value;
• and
• in response to the prediction, terminating the computation of the dot product, and outputting a 0 as the result of the activation function.