A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform actions including:
- inputting a full encoder sequence into a transformer model comprising a parameter sharing decoder pair including a first decoder being input the full encoder sequence and a first decoder input and a second decoder being input the full encoder sequence and a second decoder input, the first decoder and the second decoder performing processing independently from one another, each layer of the first decoder and each layer of the second decoder comprising a single self-attention block, a concatenation layer common to and provided at outputs of the first decoder and the second decoder, a mapping layer provided at an output of the concatenation layer, and a normalization layer provided at an output of the mapping layer;
- wherein:
- the first decoder includes N layers, each layer comprising a first masked multi-head attention block and a first feed forward network, the first masked multi-head attention block being the single self-attention block of each layer of the first decoder, wherein parameters of the first decoder are shared across all N layers of the first decoder;
- and
- the second decoder includes N layers, each layer comprising a second masked multi-head attention block and a second feed forward network, the second masked multi-head attention block being the single self-attention block of each layer of the second decoder, wherein parameters of the second decoder are shared across all N layers of the second decoder, and wherein the parameters of the first masked multi-head attention block of the first decoder are different from the parameters of the second masked multi-head attention block of the second decoder, and the parameters of the first feed forward network of the first decoder are different from the parameters of the second feed forward network of the second decoder;
- predicting, using the transformer model, sequence elements based on the full encoder sequence, the predicting including:
- predicting, using the first decoder based on the first decoder input, a first set of sequence elements based on the full encoder sequence, and
- predicting, using the second decoder based on the second decoder input, a second set of sequence elements based on the full encoder sequence, wherein the first set of sequence elements and the second set of sequence elements are the outputs of the first decoder and the second decoder;
- concatenating, using the concatenation layer common to the outputs of the first decoder and the second decoder, the first set of sequence elements and the second set of sequence elements to generate a concatenated set of sequence elements;
- applying, using the mapping layer, mapping to down-project the concatenated set of sequence elements back to an embedding size;
- providing, to the normalization layer as an input, an output of the mapping layer, an average of the first set of sequence elements, and an average of the second set of sequence elements;
- generating, using the normalization layer, an output sequence by performing processing and normalizing the provided input, wherein the performing processing and normalizing includes adding, to the output of the mapping layer, the average of the first set of sequence elements and the average of the second set of sequence elements;
- and
- outputting the output sequence which is different from the full encoder sequence.