Word-tag-based language system for sentence acceptability judgment

A non-transitory computer readable medium comprising computer executable instructions which when executed by a computer cause the computer to perform the method of:

• obtaining a word frequency distribution for a predetermined textual data set;
• determining a replacement rate based on the word frequency distribution, wherein the replacement rate is a percentage of words to be replaced with a corresponding tag;
• replacing in text of a training data set, the training data set comprising at least a portion of the predetermined textual data set, every occurrence of a word having a frequency lower than the replacement rate, with the corresponding tag to generate revised text of the training data set;
• training a plurality of bidirectional encoder representations from transformers language models with the revised text to handle cross-domain natural language processing;
• selecting a best performing trained bidirectional encoder representations from transformers language model of the plurality of trained bidirectional encoder representations from transformers language models;
• rating an acceptability of each sentence of a plurality of candidate sentences using the selected performing trained bidirectional encoder representations from transformers language model;
• and
• selecting a best sentence of the plurality of candidate sentences based on the ratings.

An apparatus comprising:

• a memory;
• and
• at least one processor, coupled to said memory, and operative to perform operations comprising:
• obtaining a word frequency distribution for a predetermined textual data set;
• determining a replacement rate based on the word frequency distribution, wherein the replacement rate is a percentage of words to be replaced with a corresponding tag;
• replacing in text of a training data set, the training data set comprising at least a portion of the predetermined textual data set, every occurrence of a word having a frequency lower than the replacement rate, with the corresponding tag to generate revised text of the training data set;
• training a plurality of bidirectional encoder representations from transformers language models with the revised text to handle cross-domain natural language processing;
• selecting a best performing trained bidirectional encoder representations from transformers language model of the plurality of trained bidirectional encoder representations from transformers language models;
• rating an acceptability of each sentence of a plurality of candidate sentences using the selected best performing trained bidirectional encoder representations from transformers language model;
• and
• selecting a best sentence of the plurality of candidate sentences based on the ratings.