Automated artificial intelligence driven readability scoring techniques

A data processing system comprising:

• a processor;
• and
• a machine-readable storage medium storing executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations comprising:
• accessing first training data for a language model that includes training data for fine-tuning a pretrained language model (PLM) to analyze textual inputs to determine readability scores for the textual inputs and to output readability scores, the PLM having already been trained on a corpus of first training data to analyze textual inputs, the PLM having not been trained to determine the readability scores for the textual inputs;
• fine-tuning training of a first instance of a PLM using the first training data to enable the first instance of the PLM to receive a textual input, to generate a readability score for the textual input, and to output the readability score for the textual input;
• receiving, from a first application on a first client device via a first network connection, a request to provide a first readability score for a first textual content from the first application;
• analyzing the first textual content with the first instance of the PLM to obtain a first readability score for the first textual content;
• and
• performing one or more actions on the first textual content based on the first readability score for the first textual content.

A data processing system comprising:

• a processor;
• and
• a machine-readable storage medium storing executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations comprising:
• receiving first textual input from a first application of a first client device;
• segmenting the first textual input into a plurality of first segments using a text first language model trained to recognize one or more segment boundaries in the first textual input and to output a plurality of first segments of textual content;
• providing the plurality of first segments to a second language model trained to analyze textual inputs and to generate a plurality of readability scores associated with the plurality of first segments, the second language model comprising a pretrained language model (PLM) which has been fine-tuned with training data to train the PLM to generate a readability score for a textual input;
• aggregating the plurality of readability scores to generate a first readability score for the first textual input;
• and
• performing one or more actions on the first textual input responsive to the first readability score falling below a readability threshold.