Unified natural language model with segmented and aggregate attention

A computer-implemented method for training a natural language model to perform multiple different types of tasks, the method comprising:

• accessing a natural language model;
• tokenizing a set of training data to generate a plurality of tokens;
• masking one or more tokens of the plurality of tokens to generate masked training data;
• dividing the set of training data into a plurality of portions;
• generating encoded representations for the plurality of portions, wherein at least some of the portions are encoded independently;
• applying an attention mechanism across the encoded representations of the plurality of portions;
• combining the encoded representations into a unified representation;
• and
• training the natural language model using the unified representation and the masked training data to enable the natural language model to perform multiple different types of tasks.

A hardware storage device that stores instructions that are executable by a processor to cause the processor to:

• access a natural language model;
• tokenize a set of training data to generate a plurality of tokens;
• mask one or more tokens of the plurality of tokens to generate masked training data;
• divide the set of training data into a plurality of portions;
• generate encoded representations for the plurality of portions, wherein at least some of the portions are encoded independently;
• apply an attention mechanism across the encoded representations of the plurality of portions;
• combine the encoded representations into a unified representation;
• and
• train the natural language model using the unified representation and the masked training data to enable the natural language model to perform multiple different types of tasks.