Masked model training of a prediction network

A method comprising:

• receiving, by a computing device, a first sequence of inputs for processing via a sub-model of a plurality of sub-models, wherein the plurality of sub-models are part of a main model;
• masking, by the computing device, an input in the first sequence of inputs with a masked value to generate a second sequence of inputs;
• processing, by the computing device, the second sequence of inputs using the sub-model to generate a sequence of features that correspond to the second sequence of inputs;
• generating, by the computing device, a first output of the main model based on the sequence of features;
• and
• training, by the computing device, the sub-model based on a feature in the sequence of features that corresponds to the masked input and the first output,
• wherein training the sub-model comprises:
• comparing the feature that corresponds to the masked input to the first sequence of inputs,
• calculating a similarity between the feature that corresponds to the masked input and the first sequence of inputs;
• and
• using the similarity to train the sub-model, wherein using the similarity to train the sub-model comprises:
• adjusting a parameter of the sub-model to increase a similarity of the feature that corresponds to the masked input to the corresponding input in the first sequence of inputs compared to a similarity for other inputs in the first sequence of inputs.

A non-transitory computer-readable storage medium having stored thereon computer executable instructions, which when executed by a computing device, cause the computing device to be operable for:

• receiving a first sequence of inputs for processing via a sub-model of a plurality of sub-models, wherein the plurality of sub-models are part of a main model;
• masking an input in the first sequence of inputs with a masked value to generate a second sequence of inputs;
• processing the second sequence of inputs using the sub-model to generate a sequence of features that correspond to the second sequence of inputs, wherein the sub-model comprises a self-attention sub-model that applies attention to inputs in the second sequence of inputs based on relationships between the inputs;
• generating a first output of the main model based on the sequence of features;
• and
• training the sub-model based on a feature in the sequence of features that corresponds to the masked input and the first output, wherein training the sub-model comprises:
• generating a first value based on a feature in the sequence of features that corresponds to the masked input;
• generating a second value based on the first output;
• and
• adjusting a parameter of the sub-model using the first value and the second value.