Generalized Sigmoids and Activation Function Learning

A system for training a neural network (NN) comprising a plurality of layers including at least one hidden layer, wherein each of the plurality of layers includes a respective set of noes, the system comprising:

• at least one memory storing computer-executable instructions;
• and
• at least one processor configured to access the at least one memory and execute the computer-executable instructions to:
• determine, for a current layer of the NN for a current iteration of the training, i) a set of inputs, the set of inputs including a set of training inputs or a set of activation results associated with the respective set of nodes of a prior layer of the NN, ii) a bias parameter, iii) a scale parameter, and iv) a respective set of weights to be applied to the set of inputs, the bias parameter, and the scale parameter for each node of the current layer of the NN;
• determine, for a particular node of the current layer during the current iteration of the training, a combined input based at least in part on the set of inputs, the respective set of weights associated with the particular node, and the bias parameter;
• weight the scale parameter using a corresponding weight in the respective set of weights to obtain a weighted scale parameter, wherein the corresponding weight is learned from one or more prior iterations of the training;
• execute, for the current iteration of the training, an activation function, wherein executing the activation function comprises applying the activation function to the combined input and the weighted scale parameter to generate an activation result for the particular node;
• determine whether the current layer is a final layer;
• and
• output, based at least in part on determining whether the current layer is a final layer, the activation result as a classifier output of the NN or providing the activation result as input to a next layer of the NN.

A computer program product for training a neural network (NN) comprising a plurality of layers including at least one hidden layer, wherein each of the plurality of layers includes a respective set of nodes, the computer program product comprising a non-transitory storage medium readable by a processing circuit, the storage medium storing instructions executable by the processing circuit to cause a method to be performed, the method comprising:

• determining, for a current layer of the NN for a current iteration of the training, i) a set of inputs, the set of inputs including a set of training inputs or a set of activation results associated with the respective set of nodes of a prior layer of the NN, ii) a bias parameter, iii) a scale parameter, and iv) a respective set of weights to be applied to the set of inputs, the bias parameter, and the scale parameter for each node of the current layer of the NN;
• determining, for a particular node of the current layer during the current iteration of the training, a combined input based at least in part on the set of inputs, the respective set of weights associated with the particular node, and the bias parameter;
• weighting the scale parameter using a corresponding weight in the respective set of weights to obtain a weighted scale parameter, wherein the corresponding weight is learned from one or more prior iterations of the training;
• executing, for the current iteration of the training, an activation function for the particular node, wherein executing the activation function comprises applying the activation function to the combined input and the weighted scale parameter to generate an activation result for the particular node;
• determining whether the current layer is a final layer;
• and
• outputting, based at least in part on determining whether the current layer is a final layer, the activation result as a classifier output of the NN or providing the activation result as input to a next layer of the NN.