Text simplification with minimal hallucination

A method for language processing, comprising:

• generating a simplified text based on a complex text;
• computing an entailment score for each sentence of the simplified text using a neural network, wherein the entailment score indicates whether the sentence of the simplified text includes information from a sentence of the complex text;
• filtering sentences of the simplified text based on the entailment score to obtain a modified text;
• computing a first hallucination score based on the complex text and the simplified text;
• computing a second hallucination score based on the complex text and the modified text;
• comparing the first hallucination score to the second hallucination score;
• and
• providing the modified text as an output based on the comparison.

An apparatus for language processing, comprising:

• a processor;
• a memory storing instructions executable by the processor;
• a neural network comprising parameters stored in the memory and configured to compute an entailment score for each sentence of a simplified text, wherein the entailment score indicates whether the sentence of the simplified text includes information from a sentence of a complex text corresponding to the sentence of the simplified text;
• a pruning component configured to generate a modified text based on the entailment score, the simplified text, and the complex text;
• and
• a hallucination scoring component configured to compute a first hallucination score based on the complex text and the simplified text and a second hallucination score based on the complex text and the modified text.