Unified natural language model with segmented and aggregate attention

A computing system that trains a natural language model to perform multiple different types of tasks rather than only a single type of task, said computing system comprising:

• one or more hardware processors;
• and
• one or more hardware storage devices that store instructions that are executable by the one or more hardware processors to cause the computing system to:
• access a natural language model;
• obtain a set of data;
• tokenize the set of data by creating tokens out of the set of data;
• mask a first subset of the tokens with a single sentinel token, resulting in generation of corrupted span data that is structured to facilitate training the natural language model to perform a natural language understanding task;
• separate the set of data into a set of chunks;
• generate encoded data from the set of chunks, wherein each chunk is encoded independently relative to one another, such that chunk-wise encoding is performed;
• apply relative attention to different encoded data in each chunk in the set of chunks, wherein applying the relative attention facilitates enabling the natural language model to perform a summarization task, wherein the relative attention is applied after the chunk-wise encoding;
• after the relative attention is applied to the set of chunks, concatenate all chunks of the set of chunks into a long representation of the encoded data that includes aggregate attention based on the relative attention applied separately to the different encoded data in each chunk, such that the long representation of the encoded data includes the aggregate attention and such that the long representation is generated after the chunk-wise encoding and after application of the relative attention to the set of chunks;
• and
• train the natural language model using the long representation of the encoded data and using the corrupted span data, resulting in the natural language model being trained to perform multiple different types of tasks.

One or more hardware storage devices that store instructions that are executable by one or more hardware processors to cause the one or more processors to:

• access a natural language model;
• obtain a set of data;
• tokenize the set of data by creating tokens out of the set of data;
• mask a first subset of the tokens with a single sentinel token, resulting in generation of corrupted span data that is structured to facilitate training the natural language model to perform a natural language understanding task;
• separate the set of data into a set of chunks;
• generate encoded data from the set of chunks, wherein each chunk is encoded independently relative to one another, such that chunk-wise encoding is performed;
• apply relative attention to different encoded data in each chunk in the set of chunks, wherein applying the relative attention facilitates enabling the natural language model to perform a summarization task, wherein the relative attention is applied after the chunk-wise encoding;
• after the relative attention is applied to the set of chunks, concatenate all chunks of the set of chunks into a long representation of the encoded data that includes aggregate attention based on the relative attention applied separately to the different encoded data in each chunk, such that the long representation of the encoded data includes the aggregate attention and such that the long representation is generated after the chunk-wise encoding and after application of the relative attention to the set of chunks;
• and
• train the natural language model using the long representation of the encoded data and using the corrupted span data, resulting in the natural language model being trained to perform multiple different types of tasks.