Neural network based representation learning for natural language processing

A non-transitory computer readable storage medium storing instructions that when executed by a computer processor, cause the computer processor to perform steps comprising:

• initializing the machine learning based model comprising a set of parameters;
• training the machine learning based model by repeatedly performing:
• generating a masked text sequence by replacing a token of an input text sequence with a masked token, for one or more tokens of the input text sequence;
• predicting one or more masked tokens of the masked text sequence using the machine learning based model;
• generating a predicted text sequence by replacing masked tokens of the masked text sequence with tokens predicted by the machine learning based model;
• using the machine learning based model as a classifier to discriminate between replaced tokens of the predicted text sequence and original tokens of the input text sequence;
• and
• adjusting the set of parameters of the machine learning based model based on an accuracy of prediction of masked tokens and an accuracy of classification of tokens using the machine learning based model;
• and
• storing the trained machine learning based model.

A computer system comprising:

• a computer processor;
• and
• a non-transitory computer readable storage medium storing instructions that when executed by the computer processor cause the computer processor to perform steps comprising:
• initializing the machine learning based model comprising a set of parameters;
• training the machine learning based model by repeatedly performing:
• generating a masked text sequence by replacing a token of an input text sequence with a masked token, for one or more tokens of the input text sequence;
• predicting one or more masked tokens of the masked text sequence using the machine learning based model;
• generating a predicted text sequence by replacing masked tokens of the masked text sequence with tokens predicted by the machine learning based model;
• using the machine learning based model as a classifier to discriminate between replaced tokens of the predicted text sequence and original tokens of the input text sequence;
• and
• adjusting the set of parameters of the machine learning based model based on an accuracy of prediction of masked tokens and an accuracy of classification of tokens using the machine learning based model;
• and
• storing the trained machine learning based model.