Universal language segment representations learning with conditional masked language model

A computing system, comprising:

• one or more processors;
• and
• one or more non-transitory computer-readable media that collectively store:
• a machine-learned language encoding model configured to process a language segment that comprises a plurality of tokens to generate a contextual language embedding for the language segment that describes the language segment in an embedding space, wherein the machine-learned language encoding model has been trained using a loss function that evaluates an ability of an additional language encoding model to perform a masked language modeling task when conditioned upon embeddings generated by the machine-learned language encoding model;
• and
• instructions that, when executed by a computing system comprising one or more computing devices, cause the computing system to perform operations, the operations comprising:
• obtaining an additional language segment that contains multiple tokens;
• using the machine-learned language encoding model to generate an embedding for the additional language segment;
• and
• performing a language task based on the embedding for the additional language segment.

One or more non-transitory computer-readable media that collectively store instructions that, when executed by a computing system comprising one or more computing devices, cause the computing system to perform operations, the operations comprising:

• processing, by the computing system, one or more sets of context data with a first machine-learned encoding model to generate a contextual embedding for the one or more sets of context data;
• generating, by the computing system, a masked version of a subject language segment, wherein the masked version of the subject language segment comprises one or more masked tokens;
• combining, by the computing system, the contextual embedding and the masked version of a subject language segment to obtain a conditioned input;
• processing, by the computing system, the conditioned input with a second machine-learned encoding model to generate one or more predictions respectively for the one or more masked tokens;
• and
• modifying, by the computing system, one or more values of one or more parameters of at least the first machine-learned encoding model based on a loss function that compares the one or more predictions respectively with the one or more masked tokens.