# US 2021/0279576 A1 · Attention neural networks with talking heads attention

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279576 A1
- Document ID: US20210279576A1
- Status: Published application
- Kind: A1
- Application number: 17/191,591
- Filed: 2021-03-03
- Published: 2021-09-09
- Earliest claimed: 2020-03-03
- Applicants: Google LLC
- Inventors: Noam M. Shazeer; Zhenzhong Lan
- Web page: https://patentgrep.com/patent/US20210279576A1

## CPC

- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0499: Feedforward networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a network input to generate a network output. In one aspect, one of the systems includes an attention neural network configured to perform the machine learning task, the attention neural network including one or more attention layers, each attention layer comprising an attention sub-layer and, optionally, a feed-forward sub-layer. At least one of the attention layers includes an attention sub-layer that applies talking heads attention instead of conventional multi-head attention.

## Claims

### Claim 1

A system for performing a machine learning task on a network input to generate a network output, the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement:
an attention neural network configured to perform the machine learning task, the attention neural network comprising a plurality of layers, each layer comprising an attention sub-layer, the attention sub-layer configured to perform operations comprising:
obtaining an input sequence for the layer comprising a respective input vector at each of one or more positions;
obtaining one or more memory vectors;
applying a plurality of query linear transformations to the input vectors to generate a plurality of sets of query vectors;
applying a plurality of key linear transformations to the memory vectors to generate a corresponding set of key vectors for each set of query vectors;
applying a plurality of value linear transformations to the memory vectors to generate a plurality of sets of value vectors;
for each query vector in each set of query vectors, generating a corresponding set of attention-logits for the query vector that includes a respective attention-logit for each key vector in the corresponding set of key vectors for the set of query vectors, comprising applying an attention function between the query vector and the set of key vectors corresponding to the set of query vectors;
generating, for each input vector and for each set of value vectors, a corresponding set of transformed attention weights that includes a respective transformed attention weight for each value vector in the set of value vectors, comprising applying an attention-logit linear transformation to the sets of attention-logits for the query vectors in the sets of query vectors;
for each input vector and each set of value vectors, computing a weighted sum of the value vectors in the set weighted by the corresponding set of transformed attention weights for the input vector to generate a weighted value vector; and
generating a respective attended vector for each input vector from the weighted value vectors for the input vector.

### Claim 2 (depends on 1)

The system of claim 1, wherein generating a respective attended vector for each input vector comprises:
applying an output linear transformation to a concatenation of the weighted value vectors for the input vector to generate the respective attended vector for the input vector.

### Claim 3 (depends on 1)

The system of claim 1, wherein the attention sub-layer is a self-attention sub-layer and wherein the one or more input vectors are the same as the one or more memory vectors.

### Claim 4 (depends on 3)

The system of claim 3, wherein the attention sub-layer is a masked self-attention sub-layer and wherein the attention function is masked.

### Claim 5 (depends on 1)

The system of claim 1, wherein generating, for each input vector and for each set of value vectors, a corresponding set of transformed attention weights comprises:
generating, from the sets of attention-logits for the query vectors in the sets of query vectors, a plurality of sets of transformed attention-logits that each include a respective transformed attention-logit for each memory vector, comprising applying the attention-logit linear transformation to the sets of attention-logits; and
for each of the sets of transformed attention-logits, generating a corresponding set of attention weights that includes a respective attention weight for each memory vector.

### Claim 6 (depends on 5)

The system of claim 5, wherein the attention-logit linear transformation is learned during the training of the attention neural network.

### Claim 7 (depends on 6)

The system of claim 6, the operations further comprising:
applying a plurality of learned input logit linear transformations to the input vectors to generate a plurality of dynamic input attention-logit matrices, and
applying a plurality of learned memory logit linear transformations to the memory vectors to generate a plurality of dynamic memory attention-logit matrices, and wherein generating the plurality of sets of transformed attention-logits that each include a respective transformed attention-logit for each memory vector further comprises:
applying the plurality of dynamic input attention-logit matrices to the sets of attention-logits, and
applying the plurality of dynamic memory attention-logit matrices to the sets of attention-logits.

### Claim 8 (depends on 5)

The system of claim 5, wherein generating, for each input vector and for each set of value vectors, a corresponding set of transformed attention weights in the set of value vectors further comprises:
generating the sets of transformed attention weights from the sets of attention weights, comprising applying an attention weight linear transformation to the sets of attention weights to generate the sets of transformed attention weights.

### Claim 9 (depends on 8)

The system of claim 8, wherein the attention weight linear transformation is learned during the training of the attention neural network.

### Claim 10 (depends on 9)

The system of claim 9, the operations further comprising:
applying a plurality of learned input attention weight linear transformations to the input vectors to generate a plurality of dynamic input attention weight matrices, and
applying a plurality of learned memory attention weight linear transformations to the memory vectors to generate a plurality of dynamic memory attention weight matrices, and wherein generating the plurality of sets of transformed attention weights comprises:
applying the plurality of dynamic input attention weight matrices to the sets of attention weights, and
applying the plurality of dynamic memory attention weight matrices to the sets of attention weights.

### Claim 11 (depends on 5)

The system of claim 5, wherein for each of the sets of transformed attention-logits, generating a corresponding set of attention weights that includes a respective attention weight for each memory vector comprises:
for each of the sets of transformed attention-logits, applying a softmax function to the transformed attention-logits in the set to generate the corresponding set of attention weights.

### Claim 12 (depends on 1)

The system of claim 1, wherein the attention function is a dot-product attention function or a scaled dot-product attention function.

### Claim 13 (depends on 1)

The system of claim 1, wherein the layer also comprises a feed-forward sub layer that is configured to:
receive an attended input sequence that includes the respective attended input vectors for each of the input vectors; and
generate an output sequence for the layer from the attended input sequence, the output sequence comprising a respective layer output vector at each of the one or more positions.

### Claim 14

A system for performing a machine learning task on a network input to generate a network output, the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement:
an attention neural network configured to perform the machine learning task, the attention neural network comprising a plurality of layers, each layer comprising an attention sub-layer, the attention sub-layer configured to perform operations comprising:
obtaining an input sequence for the layer comprising a respective input vector at each of one or more positions;
obtaining one or more memory vectors;
applying a plurality of query linear transformations to the input vectors to generate a plurality of sets of query vectors;
applying a plurality of key linear transformations to the memory vectors to generate a corresponding set of key vectors for each set of query vectors;
applying a plurality of value linear transformations to the memory vectors to generate a plurality of sets of value vectors;
for each query vector in each set of query vectors, generating a corresponding set of attention-logits for the query vector that includes a respective attention-logit for each key vector in the corresponding set of key vectors for the set of query vectors, comprising applying an attention function between the query vector and the set of key vectors corresponding to the set of query vectors;
generating, for each input vector and for each set of value vectors, a corresponding set of transformed attention weights that includes a respective transformed attention weight for each value vector in the set of value vectors, comprising applying an attention-logit linear transformation to the sets of attention-logits for the query vectors in the sets of query vectors;
for each input vector and each set of value vectors, computing a weighted sum of the value vectors in the set weighted by the corresponding set of transformed attention weights for the input vector to generate a weighted value vector; and
generating a respective attended vector for each input vector from the weighted value vectors for the input vector.

### Claim 15

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to implement:
an attention neural network configured to perform a machine learning task, the attention neural network comprising a plurality of layers, each layer comprising an attention sub-layer, the attention sub-layer configured to perform operations comprising:
obtaining an input sequence for the layer comprising a respective input vector at each of one or more positions;
obtaining one or more memory vectors;
applying a plurality of query linear transformations to the input vectors to generate a plurality of sets of query vectors;
applying a plurality of key linear transformations to the memory vectors to generate a corresponding set of key vectors for each set of query vectors;
applying a plurality of value linear transformations to the memory vectors to generate a plurality of sets of value vectors;
for each query vector in each set of query vectors, generating a corresponding set of attention-logits for the query vector that includes a respective attention-logit for each key vector in the corresponding set of key vectors for the set of query vectors, comprising applying an attention function between the query vector and the set of key vectors corresponding to the set of query vectors;
generating, for each input vector and for each set of value vectors, a corresponding set of transformed attention weights that includes a respective transformed attention weight for each value vector in the set of value vectors, comprising applying an attention-logit linear transformation to the sets of attention-logits for the query vectors in the sets of query vectors;
for each input vector and each set of value vectors, computing a weighted sum of the value vectors in the set weighted by the corresponding set of transformed attention weights for the input vector to generate a weighted value vector; and
generating a respective attended vector for each input vector from the weighted value vectors for the input vector.

### Claim 16

A method performed by one or more computers, the method comprising:
receiving a network input; and
processing the network input using an attention neural network that is configured to perform a machine learning task on the network input to generate a network output, the attention neural network comprising a plurality of layers, each layer comprising an attention sub-layer, the attention sub-layer configured to perform operations comprising:
obtaining an input sequence for the layer comprising a respective input vector at each of one or more positions;
obtaining one or more memory vectors;
applying a plurality of query linear transformations to the input vectors to generate a plurality of sets of query vectors;
applying a plurality of key linear transformations to the memory vectors to generate a corresponding set of key vectors for each set of query vectors;
applying a plurality of value linear transformations to the memory vectors to generate a plurality of sets of value vectors;
for each query vector in each set of query vectors, generating a corresponding set of attention-logits for the query vector that includes a respective attention-logit for each key vector in the corresponding set of key vectors for the set of query vectors, comprising applying an attention function between the query vector and the set of key vectors corresponding to the set of query vectors;
generating, for each input vector and for each set of value vectors, a corresponding set of transformed attention weights that includes a respective transformed attention weight for each value vector in the set of value vectors, comprising applying an attention-logit linear transformation to the sets of attention-logits for the query vectors in the sets of query vectors;
for each input vector and each set of value vectors, computing a weighted sum of the value vectors in the set weighted by the corresponding set of transformed attention weights for the input vector to generate a weighted value vector; and
generating a respective attended vector for each input vector from the weighted value vectors for the input vector.

### Claim 17 (depends on 16)

The method of claim 16, wherein generating a respective attended vector for each input vector comprises:
applying an output linear transformation to a concatenation of the weighted value vectors for the input vector to generate the respective attended vector for the input vector.

### Claim 18 (depends on 16)

The method of claim 16, wherein the attention sub-layer is a self-attention sub-layer and wherein the one or more input vectors are the same as the one or more memory vectors.

### Claim 19 (depends on 16)

The method of claim 16, wherein the attention sub-layer is a masked self-attention sub-layer and wherein the attention function is masked.

### Claim 20 (depends on 16)

The method of claim 16, wherein generating, for each input vector and for each set of value vectors, a corresponding set of transformed attention weights comprises:
generating, from the sets of attention-logits for the query vectors in the sets of query vectors, a plurality of sets of transformed attention-logits that each include a respective transformed attention-logit for each memory vector, comprising applying the attention-logit linear transformation to the sets of attention-logits; and
for each of the sets of transformed attention-logits, generating a corresponding set of attention weights that includes a respective attention weight for each memory vector.
