Back to results
US 2022/0067285 A1Published application
Learned evaluation model for grading quality of natural language generation outputs
AssigneeGoogle Inc.
InventorsThibault Sellam, Dipanjan Das, Ankur Parikh
Life of this patent
Estimating the term…
Abstract
Systems and methods for automatic evaluation of the quality of NLG outputs. In some aspects of the technology, a learned evaluation model may be pretrained first using NLG model pretraining tasks, and then with further pretraining tasks using automatically generated synthetic sentence pairs. In some cases, following pretraining, the evaluation model may be further fine-tuned using a set of human-graded sentence pairs, so that it learns to approximate the grades allocated by the human evaluators.
Claims20
A method of training a neural network, comprising:
- generating, by one or more processors of a processing system, a plurality of synthetic sentence pairs, each synthetic sentence pair of the plurality of synthetic sentence pairs comprising an original passage of text and a modified passage of text;
- generating, by the one or more processors, for each given synthetic sentence pair of the plurality of synthetic sentence pairs:
- a first training signal of a plurality of training signals based on whether the given synthetic sentence pair was generated using backtranslation;
- and
- one or more second training signals of the plurality of training signals based on a prediction from a backtranslation prediction model regarding a likelihood that one of the original passage of text or the modified passage of text of the given synthetic sentence pair could have been generated by backtranslating the other one of the original passage of text or the modified passage of text of the given synthetic sentence pair;
- pretraining, by the one or more processors, the neural network to predict, for each given synthetic sentence pair of the plurality of synthetic sentence pairs, the plurality of training signals for the given synthetic sentence pair;
- and
- fine-tuning, by the one or more processors, the neural network to predict, for each given human-graded sentence pair of a plurality of human-graded sentence pairs, a grade allocated by a human grader to the given human-graded sentence pair.
5 more elements
A processing system comprising:
- a memory;
- and
- one or more processors coupled to the memory and configured to:
- generate a plurality of synthetic sentence pairs, each synthetic sentence pair of the plurality of synthetic sentence pairs comprising an original passage of text and a modified passage of text;
- generate, for each given synthetic sentence pair of the plurality of synthetic sentence pairs:
- a first training signal of a plurality of training signals based on whether the given synthetic sentence pair was generated using backtranslation;
- and
- one or more second training signals of the plurality of training signals based on a prediction from a backtranslation prediction model regarding a likelihood that one of the original passage of text or the modified passage of text of the given synthetic sentence pair could have been generated by backtranslating the other one of the original passage of text or the modified passage of text of the given synthetic sentence pair;
- pretrain the neural network to predict, for each given synthetic sentence pair of the plurality of synthetic sentence pairs, the plurality of training signals for the given synthetic sentence pair;
- and
- fine-tune the neural network to predict, for each given human-graded sentence pair of a plurality of human-graded sentence pairs, a grade allocated by a human grader to the given human-graded sentence pair.
8 more elements
Dependencies are inferred from the claim text.