Transformer-Based Neural Network including a Mask Attention Network

A computer-implemented method for processing a sequence of data items, comprising:

• receiving, by an application, the sequence of data items based on input provided by a user via a user computing device, the application providing a transformer-based neural network having a pipeline of processing mechanisms;
• processing the sequence of data items using the transformer-based neural network to produce an output result;
• and
• providing information to the user, by the application, that is based on the output result,
• at least one processing mechanism in the pipeline of processing mechanisms including a mask attention network,
• the mask attention network including a mask data structure,
• the mask attention network performing operations of:
• computing an original attention data structure that expresses influence between pairs of data items in the sequence of data items;
• and
• computing a modified attention data structure by modifying the original attention data structure by mask values provided by the mask data structure,
• the modified attention data structure accounting for influence of neighboring data items in the sequence of data items, with respect to a particular data item under consideration, beyond that provided by the original attention data structure.

A computer-readable storage medium for storing computer-readable instructions, the computer-readable instructions, when executed by one or more hardware processors, performing a method that comprises:

• processing a sequence of data items using a mask attention network to produce a first output result,
• the mask attention network performing operations of:
• computing an original attention data structure that expresses influence between pairs of data items in the sequence of data items;
• and
• computing a modified attention data structure by modifying the original attention data structure by mask values provided by a mask data structure;
• processing the first output result using another attention network that does not use a mask data structure, to provide a second output result;
• and
• processing the second output result by a feed-forward neural network to produce a third output result,
• the mask attention network, the other attention network, and the feed-forward neural network also being implemented by the computer-readable instructions provided by the computer-readable storage medium.