Back to results
US 11,908,457 B2Granted
Orthogonally constrained multi-head attention for speech tasks
AssigneeQUALCOMM Incorporated
InventorsMingu Lee, Jinkyu Lee, Hye Jin Jang, Kyu Woong Hwang
Life of this patent
Estimating the term…
Abstract
A method for operating a neural network includes receiving an input sequence at an encoder. The input sequence is encoded to produce a set of hidden representations. Attention-heads of the neural network calculate attention weights based on the hidden representations. A context vector is calculated for each attention-head based on the attention weights and the hidden representations. Each of the context vectors correspond to a portion of the input sequence. An inference is output based on the context vectors.
Claims28
A processor-implemented method for operating a neural network, the method comprising:
- receiving an input sequence at an encoder;
- encoding the input sequence to produce a set of hidden representations of time frame t;
- providing the set of hidden representations to each of a plurality of attention heads of the neural network, each of the plurality of attention heads configured to focus on a different subset of the set of hidden representations;
- calculating corresponding attention weights for each of the different subsets of the set of hidden representations of time frame tin each attention head of the plurality of attention-heads based on a corresponding score, each score being computed based on a nonlinear scoring function of the set of hidden representations of time frame t and learnable parameters;
- calculating a context vector for each attention-head based on the attention weights of time frame t and the set of hidden representations of time frame t, each context vector corresponding to a portion of the input sequence;
- and
- outputting an inference based on the context vectors.
4 more elements
An apparatus for operating a neural network, the apparatus comprising:
- a memory configured to store one or more parameters of the neural network;
- and
- at least one processor coupled to the memory, the at least one processor being configured:
- to receive an input sequence at an encoder;
- to encode the input sequence to produce a set of hidden representations of time frame t;
- to provide the set of hidden representations to each of a plurality of attention heads of the neural network, each of the plurality of attention heads configured to focus on a different subset of the set of hidden representations;
- to calculate corresponding attention weights for each of the different subsets of the set of hidden representations of time frame tin each attention head of the plurality of attention-heads based on a corresponding score, each score being computed based on a nonlinear scoring function of the set of hidden representations of time frame t and learnable parameters, the learnable parameters being stored in the memory;
- to calculate a context vector for each attention-head based on the attention weights of time frame t and the set of hidden representations of time frame t, each context vector corresponding to a portion of the input sequence;
- and
- to output an inference based on the context vectors.
7 more elements
An apparatus for operating a neural network, the apparatus comprising:
- means for receiving an input sequence at an encoder;
- means for encoding the input sequence to produce a set of hidden representations of time frame t;
- means for providing the set of hidden representations to each of a plurality of attention heads of the neural network, each of the plurality of attention heads configured to focus on a different subset of the set of hidden representations;
- means for calculating corresponding attention weights for each of the different subsets of the set of hidden representations of time frame tin each attention head of the plurality of attention-heads based on a corresponding score, each score being computed based on a nonlinear scoring function of the set of hidden representations of time frame t and learnable parameters;
- means for calculating a context vector for each attention-head based on the attention weights of time frame t and the set of hidden representations of time frame t, each context vector corresponding to a portion of the input sequence;
- and
- means for outputting an inference based on the context vectors.
4 more elements
A non-transitory computer readable medium having encoded thereon program code for operating a neural network, the program code being executed by a processor and comprising:
- program code to receive an input sequence at an encoder;
- program code to separate the input sequence into sequence parts;
- program code to encode the sequence parts to produce a set of hidden representations of time frame t;
- program code to provide the set of hidden representations to each of a plurality of attention heads of the neural network, each of the plurality of attention heads configured to focus on a different subset of the set of hidden representations;
- program code to calculate corresponding attention weights for each of the different subsets of the set of hidden representations of time frame tin each attention head of the plurality of attention-heads based on a corresponding score, each score being computed based on a nonlinear scoring function of the set of hidden representations of time frame t and learnable parameters;
- program code to calculate a context vector for each attention-head based on the attention weights of time frame t and the set of hidden representations of time frame t, each context vector corresponding to a portion of the input sequence;
- and
- program code to output an inference based on the context vectors.
5 more elements
Dependencies are inferred from the claim text.