# US 11,908,457 B2 · Orthogonally constrained multi-head attention for speech tasks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,457 B2
- Document ID: US11908457B2
- Status: Granted
- Kind: B2
- Application number: 16/920,519
- Filed: 2020-07-03
- Published: 2021-01-07
- Granted: 2024-02-20
- Earliest claimed: 2019-07-03
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Mingu Lee; Jinkyu Lee; Hye Jin Jang; Kyu Woong Hwang
- Web page: https://patentgrep.com/patent/US11908457B2

## CPC

- G10L 15/16: using artificial neural networks
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/048: Activation functions
- G06N 3/08: Learning methods
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/045: Combinations of networks
- G06N 3/047: Probabilistic or stochastic networks
- G10L 2015/088: {Word spotting}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method for operating a neural network includes receiving an input sequence at an encoder. The input sequence is encoded to produce a set of hidden representations. Attention-heads of the neural network calculate attention weights based on the hidden representations. A context vector is calculated for each attention-head based on the attention weights and the hidden representations. Each of the context vectors correspond to a portion of the input sequence. An inference is output based on the context vectors.

## Claims

### Claim 1

A processor-implemented method for operating a neural network, the method comprising:
receiving an input sequence at an encoder;
encoding the input sequence to produce a set of hidden representations of time frame t;
providing the set of hidden representations to each of a plurality of attention heads of the neural network, each of the plurality of attention heads configured to focus on a different subset of the set of hidden representations;
calculating corresponding attention weights for each of the different subsets of the set of hidden representations of time frame tin each attention head of the plurality of attention-heads based on a corresponding score, each score being computed based on a nonlinear scoring function of the set of hidden representations of time frame t and learnable parameters;
calculating a context vector for each attention-head based on the attention weights of time frame t and the set of hidden representations of time frame t, each context vector corresponding to a portion of the input sequence; and
outputting an inference based on the context vectors.

### Claim 2 (depends on 1)

The processor-implemented method of claim 1, in which the input sequence comprises an acoustic feature and the inference includes an indication of whether a keyword is included in a corresponding portion of the input sequence.

### Claim 3 (depends on 1)

The processor-implemented method of claim 1, in which each of the context vectors are orthogonal to other context vectors of other attention-heads.

### Claim 4 (depends on 1)

The processor-implemented method of claim 1, in which the set of scores are computed based on a scalar projection.

### Claim 5 (depends on 1)

The processor-implemented method of claim 1, further comprising selectively regularizing the neural network based on orthogonality constraints of the attention weights and the context vectors.

### Claim 6 (depends on 5)

The processor-implemented method of claim 5, in which the selectively regularizing further comprises calculating regularization terms only for positive samples in the input sequence.

### Claim 7 (depends on 1)

The processor-implemented method of claim 1, in which the nonlinear scoring function is computed based on a scalar projection, a weight and a bias, where the learnable parameters include the scalar projection, the weight and the bias.

### Claim 8 (depends on 1)

The processor-implemented method of claim 1, further comprising applying a softmax function to each of the context vectors to compute a posterior probability of a keyword state being in a corresponding portion of the input sequence.

### Claim 9 (depends on 8)

The processor-implemented method of claim 8, further comprising detecting a keyword responsive to the posterior probability of the keyword state being in a corresponding portion of the input sequence is above a predefined threshold.

### Claim 10

An apparatus for operating a neural network, the apparatus comprising:
a memory configured to store one or more parameters of the neural network; and
at least one processor coupled to the memory, the at least one processor being configured:
to receive an input sequence at an encoder;
to encode the input sequence to produce a set of hidden representations of time frame t;
to provide the set of hidden representations to each of a plurality of attention heads of the neural network, each of the plurality of attention heads configured to focus on a different subset of the set of hidden representations;
to calculate corresponding attention weights for each of the different subsets of the set of hidden representations of time frame tin each attention head of the plurality of attention-heads based on a corresponding score, each score being computed based on a nonlinear scoring function of the set of hidden representations of time frame t and learnable parameters, the learnable parameters being stored in the memory;
to calculate a context vector for each attention-head based on the attention weights of time frame t and the set of hidden representations of time frame t, each context vector corresponding to a portion of the input sequence; and
to output an inference based on the context vectors.

### Claim 11 (depends on 10)

The apparatus of claim 10, in which the input sequence comprises an acoustic feature and the at least one processor is further configured to output an inference indicating whether a keyword is included in a corresponding portion of the input sequence.

### Claim 12 (depends on 10)

The apparatus of claim 10, in which each of the context vectors are orthogonal to other context vectors of other attention-heads.

### Claim 13 (depends on 10)

The apparatus of claim 10, in which the at least one processor is further configured to compute the set of scores based on a scalar projection.

### Claim 14 (depends on 10)

The apparatus of claim 10, in which the at least one processor is further configured to selectively regularize the neural network based on orthogonality constraints of the attention weights and the context vectors.

### Claim 15 (depends on 14)

The apparatus of claim 14, in which the at least one processor is further configured to calculate regularization terms only for positive samples in the input sequence.

### Claim 16 (depends on 10)

The apparatus of claim 10, in which the nonlinear scoring function is computed based on a scalar projection, a weight and a bias, where the learnable parameters include the scalar projection, the weight and the bias.

### Claim 17

An apparatus for operating a neural network, the apparatus comprising:
means for receiving an input sequence at an encoder;
means for encoding the input sequence to produce a set of hidden representations of time frame t;
means for providing the set of hidden representations to each of a plurality of attention heads of the neural network, each of the plurality of attention heads configured to focus on a different subset of the set of hidden representations;
means for calculating corresponding attention weights for each of the different subsets of the set of hidden representations of time frame tin each attention head of the plurality of attention-heads based on a corresponding score, each score being computed based on a nonlinear scoring function of the set of hidden representations of time frame t and learnable parameters;
means for calculating a context vector for each attention-head based on the attention weights of time frame t and the set of hidden representations of time frame t, each context vector corresponding to a portion of the input sequence; and
means for outputting an inference based on the context vectors.

### Claim 18 (depends on 17)

The apparatus of claim 17, in which the input sequence comprises an acoustic feature and further comprising means for outputting an inference indicating whether a keyword is included in a corresponding portion of the input sequence.

### Claim 19 (depends on 17)

The apparatus of claim 17, in which each of the context vectors are orthogonal to other context vectors of other attention-heads.

### Claim 20 (depends on 17)

The apparatus of claim 17, further comprising means for computing the set of scores based on a scalar projection.

### Claim 21 (depends on 17)

The apparatus of claim 17, further comprising means for selectively regularizing the neural network based on orthogonality constraints of the attention weights and the context vectors.

### Claim 22 (depends on 21)

The apparatus of claim 21, further comprising means for calculating regularization terms only for positive samples in the input sequence.

### Claim 23 (depends on 17)

The apparatus of claim 17, in which the at least one processor is further configured to detect a keyword responsive to the posterior probability of the keyword state being in a corresponding portion of the input sequence is above a predefined threshold.

### Claim 24 (depends on 17)

The apparatus of claim 17, in which the attention heads are configured to perform an speech recognition task, a speaker verification task or a keyword spotting task.

### Claim 25

A non-transitory computer readable medium having encoded thereon program code for operating a neural network, the program code being executed by a processor and comprising:
program code to receive an input sequence at an encoder;
program code to separate the input sequence into sequence parts;
program code to encode the sequence parts to produce a set of hidden representations of time frame t;
program code to provide the set of hidden representations to each of a plurality of attention heads of the neural network, each of the plurality of attention heads configured to focus on a different subset of the set of hidden representations;
program code to calculate corresponding attention weights for each of the different subsets of the set of hidden representations of time frame tin each attention head of the plurality of attention-heads based on a corresponding score, each score being computed based on a nonlinear scoring function of the set of hidden representations of time frame t and learnable parameters;
program code to calculate a context vector for each attention-head based on the attention weights of time frame t and the set of hidden representations of time frame t, each context vector corresponding to a portion of the input sequence; and
program code to output an inference based on the context vectors.

### Claim 26 (depends on 25)

The non-transitory computer readable medium of claim 25, in which the input sequence comprises an acoustic feature and further comprising program code to output an inference indicating whether a keyword is included in a corresponding portion of the input sequence.

### Claim 27 (depends on 25)

The non-transitory computer readable medium of claim 25, in which each of the context vectors are orthogonal to other context vectors of other attention-heads.

### Claim 28 (depends on 25)

The non-transitory computer readable medium of claim 25, further comprising program code to selectively regularize the neural network based on orthogonality constraints of the attention weights and the context vectors.