Attention neural networks with talking heads attention
OwnerNoam M. Shazeer, Zhenzhong Laninventors
Life of this patent
Estimating the term…
Loading patent…
OwnerNoam M. Shazeer, Zhenzhong Laninventors
Estimating the term…
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a network input to generate a network output. In one aspect, one of the systems includes an attention neural network configured to perform the machine learning task, the attention neural network including one or more attention layers, each attention layer comprising an attention sub-layer and, optionally, a feed-forward sub-layer. At least one of the attention layers includes an attention sub-layer that applies talking heads attention instead of conventional multi-head attention.
A system for performing a machine learning task on a network input to generate a network output, the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement:
A system for performing a machine learning task on a network input to generate a network output, the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement:
One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to implement:
A method performed by one or more computers, the method comprising:
Dependencies are inferred from the claim text.