Layer normalization for calibrated uncertainty in deep learning

A computer implemented method for layer normalization in machine learning applications comprising:

• performing a training sequence that includes sampling a random set of activations corresponding to fix fraction of overall activations to provide a plurality of subsampled activations, computing an average across the plurality of subsampled activations, computing a standard deviation across the plurality of subsampled activations, and employing two statistics including the average of the subsampled activations and the standard deviation across the plurality of subsampled activations to normalize all of the activations as a layer normalization;
• and
• performing an evaluation sequence employing a Monte Carlo approximation that includes using a subsample input feature for calculation of normalization statics, and computing the average of running a network forward multiple times with the layer normalization from the training sequence, wherein the average is an output providing a prediction.

A computer implemented method is provided for layer normalization in machine learning applications comprising:

• performing a training sequence that includes sampling a random set of activations corresponding to fix fraction of overall activations to provide a plurality of subsampled activations, computing an average across the plurality of subsampled activations, computing a standard deviation across the plurality of subsampled activations, and employing two statistics including the average of the subsampled activations and the standard deviation across the plurality of subsampled activations to normalize all of the activations as a layer normalization;
• and
• performing an evaluation sequence employing a sampling procedure that translates to a Monte Carlo layer normalization of the training sequence, and employing a Laplace approximation to provide a prediction without subsampling of input features.