Incorporating alignment into sequence generation neural networks

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output with a corresponding alignment that defines how the output relates to the model input. In one aspect, a system comprises receiving a model input, processing the model input to generate an input sequence of input tokens that represent the model input, generating, by processing the input sequence of input tokens using a sequence generation neural network, a combined output sequence of tokens comprising alignment tokens and output tokens, wherein each alignment token encodes an alignment between at least one of the input tokens and one or more of the output tokens according to an alignment mapping encoding, and generating an output comprising one or more output elements by decoding at least the output tokens.

Dependencies are inferred from the claim text.