# US 2025/0094784 A1 · Layer normalization for calibrated uncertainty in deep learning > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2025/0094784 A1 - Document ID: US20250094784A1 - Status: Published application - Kind: A1 - Application number: 18/470,714 - Filed: 2023-09-20 - Published: 2025-03-20 - Earliest claimed: 2023-09-20 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Thomas Frick; Mattia Rigotti; Diego Matteo Antognini; Ioana Giurgiu; Adelmo Cristiano Innocenza Malossi - Web page: https://patentgrep.com/patent/US20250094784A1 ## CPC - G06N 3/047: Probabilistic or stochastic networks - G06N 3/08: Learning methods - G06N 3/048: Activation functions - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Layer normalization in machine learning applications that includes sampling a random set of activations corresponding to a fix fraction of the overall activations to provide a plurality of subsampled activations; computing the average across the plurality of subsampled activations; and computing the standard deviation across the plurality of subsampled activations. The layer normalization further includes employing two statistics including the average of the subsampled activations and the standard deviation across the plurality of subsampled activations to normalize all of the activations as a layer normalization. ## Claims ### Claim 1 A computer implemented method is for layer normalization in machine learning applications comprising: sampling a random set of activations corresponding to fixed fraction of overall activations to provide a plurality of subsampled activations; computing an average across the plurality of subsampled activations; computing a standard deviation across the plurality of subsampled activations; and employing two statistics including the average of the subsampled activations and the standard deviation across the plurality of subsampled activations to normalize all of the activations as a layer normalization. ### Claim 2 (depends on 1) The computer implemented method of claim 1, wherein the sampling of the random set of activations comprises selecting a set of neurons of a neural network having preactivations from a total number of neurons in the neural network. ### Claim 3 (depends on 1) The computer implemented method of claim 1, wherein the computing the average and the standard deviation from the plurality of subsampled activations to normalize all of the activations as the layer normalization comprises: μ ˜ l = 1 ❘ "\[LeftBracketingBar]" S l ❘ "\[RightBracketingBar]" ⁢ ∑ i ∈ S l a i l , ( σ ˜ l ) 2 = 1 ❘ "\[LeftBracketingBar]" S l ❘ "\[RightBracketingBar]" ⁢ ∑ i ∈ S l ( a i l - μ ˜ l ) 2 , a ~ i l = ( a i l - μ ˜ l ) σ ˜ l wherein the subsampled activations is equal to: S1⊂[Nl]={1, . . . , Nl} the average from the plurality of subsampled activations is equal to {tilde over (μ)}l′, the standard deviation from the plurality of subsampled activations is equal to {tilde over (σ)}l, and the layer normalization is a ~ i l = ( a i l - μ ~ l ) σ ~ l , in which α is activations in a layer of a neural network. ### Claim 4 (depends on 1) The computer implemented method of claim 1, wherein the layer normalization is a step of a training sequence for a neural network used in machine learning of a model that controls a vehicle in an autonomous vehicle application. ### Claim 5 (depends on 1) The computer implemented method of claim 1, wherein the layer normalization is a step of a training sequence for a neural network used in machine learning of a model in an object identification application. ### Claim 6 A computer implemented method for layer normalization in machine learning applications comprising: performing a training sequence that includes sampling a random set of activations corresponding to fix fraction of overall activations to provide a plurality of subsampled activations, computing an average across the plurality of subsampled activations, computing a standard deviation across the plurality of subsampled activations, and employing two statistics including the average of the subsampled activations and the standard deviation across the plurality of subsampled activations to normalize all of the activations as a layer normalization; and performing an evaluation sequence employing a Monte Carlo approximation that includes using a subsample input feature for calculation of normalization statics, and computing the average of running a network forward multiple times with the layer normalization from the training sequence, wherein the average is an output providing a prediction. ### Claim 7 (depends on 6) The computer implemented method of claim 6, wherein the sampling a random set of activations comprises selecting a set of neurons of a neural network having preactivations from a total number of neurons in the neural network. ### Claim 8 (depends on 6) The computer implemented method of claim 6, wherein computing the average and the standard deviation from the plurality of subsampled activations to normalize all of the activations as the layer normalization comprises: μ ˜ l = 1 ❘ "\[LeftBracketingBar]" S l ❘ "\[RightBracketingBar]" ⁢ ∑ i ∈ S l a i l , ( σ ˜ l ) 2 = 1 ❘ "\[LeftBracketingBar]" S l ❘ "\[RightBracketingBar]" ⁢ ∑ i ∈ S l ( a i l - μ ˜ l ) 2 , a ~ i l = ( a i l - μ ˜ l ) σ ˜ l wherein the subsampled activations is equal to: Sl⊂[Nl]={1, . . . , Nl}, the average from the plurality of subsampled activations is equal to {tilde over (μ)}l, the standard deviation from the plurality of subsampled activations is equal to {tilde over (σ)}l, and the layer normalization is a ~ i l = ( a i l - μ ~ l ) σ ~ l , in which a˜1. ### Claim 9 A computer implemented method is provided for layer normalization in machine learning applications comprising: performing a training sequence that includes sampling a random set of activations corresponding to fix fraction of overall activations to provide a plurality of subsampled activations, computing an average across the plurality of subsampled activations, computing a standard deviation across the plurality of subsampled activations, and employing two statistics including the average of the subsampled activations and the standard deviation across the plurality of subsampled activations to normalize all of the activations as a layer normalization; and performing an evaluation sequence employing a sampling procedure that translates to a Monte Carlo layer normalization of the training sequence, and employing a Laplace approximation to provide a prediction without subsampling of input features. ### Claim 10 (depends on 9) The computer implemented method of claim 9, wherein the sampling of the random set of the activations comprises selecting a set of neurons of a neural network having preactivations from a total number of neurons in the neural network. ### Claim 11 (depends on 9) The computer implemented method of claim 9, wherein the computing the average and the standard deviation from the plurality of subsampled activations to normalize all of the activations as the layer normalization comprises: μ ˜ l = 1 ❘ "\[LeftBracketingBar]" S l ❘ "\[RightBracketingBar]" ⁢ ∑ i ∈ S l a i l , ( σ ˜ l ) 2 = 1 ❘ "\[LeftBracketingBar]" S l ❘ "\[RightBracketingBar]" ⁢ ∑ i ∈ S l ( a i l - μ ˜ l ) 2 , a ~ i l = ( a i l - μ ˜ l ) σ ˜ l wherein the subsampled activations is equal to: S1⊂[Nl]={1, . . . , Nl}, the average from the plurality of subsampled activations is equal to {tilde over (μ)}l, the standard deviation from the plurality of subsampled activations is equal to {tilde over (σ)}l, and the layer normalization is a ~ i l = ( a i l - μ ~ l ) σ ~ l , in which a˜1. ### Claim 12 (depends on 9) The computer implemented method of claim 9, wherein the Laplace approximation is equal to: 1 N ⁢ ∑ N n = 1 p θ ˆ n ( y ⁢ ❘ "\[LeftBracketingBar]" x ) ⁢ with ⁢ θ ˆ n ∼ q 0 ( θ ˆ ) = 1 N ⁢ ∑ N n = 1 softmax ( f θ ˆ n ( x ) ) ≈ softmax ( f θ ¯ ( x ) ) . ### Claim 13 A system for layer normalization in machine learning applications including a hardware processor; and a memory that stores a computer program product, the computer program product of the system includes instructions comprising: sample, with the hardware processor, a random set of activations corresponding to fix fraction of overall activations to provide a plurality of subsampled activations; compute, with the hardware processor, an average across the plurality of subsampled activations; compute, using the hardware processor, a standard deviation across the plurality of subsampled activations; and employ, using the hardware processor, two statistics including the average of the subsampled activations and the standard deviation across the plurality of subsampled activations to normalize all of the activations as a layer normalization. ### Claim 14 (depends on 13) The system of claim 13 further comprising to perform with the hardware processor an evaluation sequence employing a Monte Carlo approximation that includes using a subsample input feature for calculation of normalization statics, and computing the average of running a network forward multiple times with the layer normalization, wherein the average is an output providing a prediction. ### Claim 15 (depends on 13) The system of claim 13 further comprising to perform, with the hardware processor, an evaluation sequence employing a sampling procedure that translates to a Monte Carlo layer normalization, and employing a Laplace approximation to provide a prediction without subsampling of input features. ### Claim 16 (depends on 13) The system of claim 13, wherein the sampling of the random set of activations comprises selecting a set of neurons of a neural network having preactivations from a total number of neurons in the neural network. ### Claim 17 (depends on 13) The system of claim 13, wherein the computing the average and the standard deviation from the plurality of subsampled activations to normalize all of the activations as the layer normalization comprises: μ ˜ l = 1 ❘ "\[LeftBracketingBar]" S l ❘ "\[RightBracketingBar]" ⁢ ∑ i ∈ S l a i l , ( σ ˜ l ) 2 = 1 ❘ "\[LeftBracketingBar]" S l ❘ "\[RightBracketingBar]" ⁢ ∑ i ∈ S l ( a i l - μ ˜ l ) 2 , a ~ i l = ( a i l - μ ˜ l ) σ ˜ l wherein the subsampled activations is equal to: Sl⊂[Nl]={1, . . . , Nl}, the average from the plurality of subsampled activations is equal to {tilde over (μ)}l, the standard deviation from the plurality of subsampled activations is equal to {tilde over (σ)}l, and the layer normalization is a ~ i l = ( a i l - μ ˜ l ) σ ˜ l , in which a˜1. ### Claim 18 (depends on 13) The system of claim 13, wherein the layer normalization is a step of a training sequence for a neural network used in machine learning of a model that controls a vehicle in an autonomous vehicle application. ### Claim 19 (depends on 13) The system of claim 13, wherein the layer normalization is a step of a training sequence for a neural network used in machine learning of a model in an object identification application. ### Claim 20 A computer program product for layer normalization in machine learning applications, the computer program product including a computer readable storage medium having computer readable program code embodied therewith, the program code executable by a hardware processor to cause the hardware processor to: sample a random set of activations corresponding to fix fraction of an overall activations to provide a plurality of subsampled activations; compute an average across the plurality of subsampled activations; compute a standard deviation across the plurality of subsampled activations; and employ two statistics including the average of the subsampled activations and the standard deviation across the plurality of subsampled activations to normalize all of the activations as a layer normalization. ### Claim 21 (depends on 20) The computer program product of claim 20 further comprising to perform with the hardware processor an evaluation sequence employing a Monte Carlo approximation that includes using a subsample input feature for calculation of normalization statics, and computing the average of running a network forward multiple times with the layer normalization, wherein the average is an output providing a prediction. ### Claim 22 (depends on 20) The computer program product of claim 20 further comprising to perform, with the hardware processor, an evaluation sequence employing a sampling procedure that translates to a Monte Carlo layer normalization, and employing a Laplace approximation to provide a prediction without subsampling of input features. ### Claim 23 (depends on 20) The computer program product of claim 20, wherein the computing the average and the standard deviation from the plurality of subsampled activations to normalize all of the activations as the layer normalization comprises: μ ˜ l = 1 ❘ "\[LeftBracketingBar]" S l ❘ "\[RightBracketingBar]" ⁢ ∑ i ∈ S l a i l , ( σ ˜ l ) 2 = 1 ❘ "\[LeftBracketingBar]" S l ❘ "\[RightBracketingBar]" ⁢ ∑ i ∈ S l ( a i l - μ ˜ l ) 2 , a ~ i l = ( a i l - μ ˜ l ) σ ˜ l wherein the subsampled activations is equal to: Sl=[Nl]={1, . . . , Nl}, the average from the plurality of subsampled activations is equal to {tilde over (μ)}l, the standard deviation from the plurality of subsampled activations is equal to {tilde over (σ)}l, and the layer normalization is a ~ i l = ( a i l - μ ˜ l ) σ ˜ l , in which a˜1. ### Claim 24 (depends on 20) The computer program product of claim 20, wherein the layer normalization is a step of a training sequence for a neural network used in machine learning of a model that controls a vehicle in an autonomous vehicle application. ### Claim 25 (depends on 20) The computer program product of claim 20, wherein the layer normalization is a step of a training sequence for a neural network used in machine learning of a model in an object identification application.