Attention neural networks with linear units

A method performed by one or more computers, the method comprising:

• receiving a network input;
• and
• processing the network input using an attention neural network to generate a network output for a machine learning task, the attention neural network comprising a plurality of layers, each layer comprising an attention sub-layer and a feed-forward sub-layer, the attention sub-layer configured to:
• receive an input sequence for the attention layer comprising a respective layer input at each of one or more positions;
• and
• generate an attended input sequence at least in part by applying an attention mechanism to the input sequence for the attention layer, the attended input sequence comprising a respective attended layer input at each of the one or more positions, and the feed-forward sub-layer configured to:
• receive the attended input sequence;
• and
• generate an output sequence for the attention layer from the attended input sequence, the output sequence comprising a respective layer output at each of the one or more positions, and the generating comprising, for each position of the one or more positions:
• generating a first linear transformation output by applying a first linear transformation to the attended layer input at the position;
• generating a first transformed input by applying a non-linear activation function to the first linear transformation output;
• generating a second transformed input by applying a second linear transformation to the attended layer input at the position;
• generating a third transformed input by performing an element-wise multiplication between the first transformed input and the second transformed input;
• and
• generating the layer output at the position based on the third transformed input,
• wherein
• the first and second linear transformations have been learned during training of the attention neural network to perform the machine learning task,
• the training of the attention neural network comprises training the attention neural network on an unsupervised data set through unsupervised learning,
• the same first linear transformation is applied to the attended layer inputs at each of the positions in the input sequence,
• the same second linear transformation is applied to the attended layer inputs at each of the positions in the input sequence, and
• the attention neural network further comprises one or more output layers that are configured to process at least one of the layer outputs in the output sequence generated by one of the plurality of attention layers to generate at least a portion of the network output for the machine learning task.

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to implement:

• an attention neural network configured to process a network input to generate a network output for a machine learning task, the attention neural network comprising a plurality of attention layers, each attention layer comprising an attention sub-layer and a feed-forward sub-layer, the attention sub-layer configured to:
• receive an input sequence for the attention layer comprising a respective layer input at each of one or more positions;
• and
• generate an attended input sequence at least in part by applying an attention mechanism to the input sequence for the attention layer, the attended input sequence comprising a respective attended layer input at each of the one or more positions, and the feed-forward sub-layer configured to:
• receive the attended input sequence;
• and
• generate an output sequence for the attention layer from the attended input sequence, the output sequence comprising a respective layer output at each of the one or more positions, and the generating comprising, for each position of the one or more positions:
• generating a first linear transformation output by applying a first linear transformation to the attended layer input at the position;
• generating a first transformed input by applying a non-linear activation function to the first linear transformation output;
• generating a second transformed input by applying a second linear transformation to the attended layer input at the position;
• generating a third transformed input by performing an element-wise multiplication between the first transformed input and the second transformed input;
• and
• generating the layer output at the position based on the third transformed input,
• wherein
• the first and second linear transformations have been learned during training of the attention neural network to perform the machine learning task,
• the training of the attention neural network comprises training the attention neural network on an unsupervised data set through unsupervised learning,
• the same first linear transformation is applied to the attended layer inputs at each of the positions in the input sequence,
• the same second linear transformation is applied to the attended layer inputs at each of the positions in the input sequence, and
• the attention neural network further comprises one or more output layers that are configured to process at least one of the layer outputs in the output sequence generated by one of the plurality of attention layers to generate at least a portion of the network output for the machine learning task.