Back to results
US 12,223,273 B2Granted
Learned evaluation model for grading quality of natural language generation outputs
AssigneeGoogle Inc.
InventorsThibault Sellam, Dipanjan Das, Ankur Parikh
Life of this patent
Estimating the term…
Abstract
Systems and methods for automatic evaluation of the quality of NLG outputs. In some aspects of the technology, a learned evaluation model may be pretrained first using NLG model pretraining tasks, and then with further pretraining tasks using automatically generated synthetic sentence pairs. In some cases, following pretraining, the evaluation model may be further fine-tuned using a set of human-graded sentence pairs, so that it learns to approximate the grades allocated by the human evaluators.
Claims20
A method of training a neural network, comprising:
- generating, by one or more processors, a first training signal of a plurality of training signals based on whether a given synthetic sentence pair was generated using backtranslation;
- and
- generating, by the one or more processors, one or more second training signals of the plurality of training signals based on a prediction from a textual entailment model regarding a likelihood that a modified passage of text of the given synthetic sentence pair entails or contradicts an original passage of text of the given synthetic sentence pair;
- pretraining, by the one or more processors, the neural network to predict the plurality of training signals for the given synthetic sentence pair;
- and
- fine-tuning, by the one or more processors, the neural network to predict a grade allocated to a graded sentence pair.
3 more elements
A processing system comprising:
- memory;
- and
- one or more processors operatively coupled to the memory and configured to:
- generate a first training signal of a plurality of training signals based on whether a given synthetic sentence pair was generated using backtranslation;
- and
- generate one or more second training signals of the plurality of training signals based on a prediction from a textual entailment model regarding a likelihood that a modified passage of text of the given synthetic sentence pair entails or contradicts an original passage of text of the given synthetic sentence pair;
- pretrain a neural network to predict the plurality of training signals for the given synthetic sentence pair;
- and
- fine-tune the neural network to predict a grade allocated to a graded sentence pair.
6 more elements
Dependencies are inferred from the claim text.