Synthetic ground truth expansion
AssigneeWelltok, Inc.
InventorsMatthew Kellar MacLeod, Jacque W. Swartz, Marissa Victoria Ponder
Life of this patent
Estimating the term…
Abstract
A ground truth expansion system that generates an expanded set of synthetic questions and selects a targeted subset of questions for machine learning training. The machine learning may be used to train an automated inquiry system that responds to questions received from individuals about subject matter of interest. The automated inquiry system is particularly suitable for use in, for example, responding to questions raised by insured individuals about their healthcare benefits.
Claims16
A computer-implemented method for generating a ground truth for training a text classifier, the method comprising:
- retrieving a plurality of question forms and a plurality of intent forms, wherein each intent form includes an intent identifier, at least one intent synonyms, and answer information;
- generating, based on the question forms and intent forms, an expanded question set comprised of question text and corresponding answers, wherein each question text-answer pair is associated with a topic;
- constructing vectors characterizing the questions of the expanded question set by, for each question text:
- dividing the question text into one or more text segments;
- calculating metrics of significance for the one or more text segments;
- and
- generating a vector associated with the question text comprised of the metrics of significance for the one or more text segments;
- determining a number of question text-answer pairs needed to train a text classifier based on a topic associated with the text classifier;
- generating a ground truth training set for training the text classifier by:
- identifying question text-answer pairs from the expanded question set associated with the topic of the text classifier;
- calculating vector distances between the vector characterizations of the identified question text-answer pairs;
- and
- selecting, based on the vector distances, question text-answer pairs for the ground truth training set based on the determined number needed to train the text classifier;
- and
- training the text classifier using the ground truth training set.
A non-transitory computer-readable medium containing instruction configured to cause one or more processors to perform a method of generating a ground truth for training a text classifier, the method comprising:
- retrieving a plurality of question forms and a plurality of intent forms, wherein each intent form includes an intent identifier, at least one intent synonyms, and answer information;
- generating, based on the question forms and intent forms, an expanded question set comprised of question text and corresponding answers, wherein each question text-answer pair is associated with a topic;
- constructing vectors characterizing the questions of the expanded question set by, for each question text:
- dividing the question text into one or more text segments;
- calculating metrics of significance for the one or more text segments;
- and
- generating a vector associated with the question text comprised of the metrics of significance for the one or more text segments;
- determining a number of question text-answer pairs needed to train a text classifier based on a topic associated with the text classifier;
- generating a ground truth training set for training the text classifier by:
- identifying question text-answer pairs from the expanded question set associated with the topic of the text classifier;
- calculating vector distances between the vector characterizations of the identified question text-answer pairs;
- and
- selecting, based on the vector distances, question text-answer pairs for the ground truth training set based on the determined number needed to train the text classifier;
- and
- training the text classifier using the ground truth training set.
Dependencies are inferred from the claim text.