Harvesting question/answer training data from watched hypotheses in a deep qa system

A computer program product for harvesting training data for a training set for use by a system capable of answering questions, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:

• receive, from a user, an input question;
• process the input question and returning, to the user, a result set comprising one or more ranked hypotheses and one or more ranked evidence passages corresponding to the one or more ranked hypotheses;
• receive, from the user, an indication that one of the one or more ranked hypotheses is to be designated a watched hypothesis;
• add the input question and the watched hypothesis to a to-be-vetted question/answer (QA) pair set comprising one or more to-be-vetted QA pairs;
• vet each of the one or more to-be-vetted QA pairs in the to-be-vetted QA pair set through a first-pass automatic vetting procedure;
• if a vetted QA pair passes the first-pass automatic vetting procedure, add the vetted QA pair to the training set;
• and
• retrain one or more ranking models based upon the training set.

A system for harvesting training data for a training set, the system comprising:

• a training data harvesting processor configured to:
• receive, from a user, an input question;
• process the input question and returning, to the user, a result set comprising one or more ranked hypotheses and one or more ranked evidence passages corresponding to the one or more ranked hypotheses;
• receive, from the user, an indication that one of the one or more ranked hypotheses is to be designated a watched hypothesis;
• add the input question and the watched hypothesis to a to-be-vetted question/answer (QA) pair set comprising one or more to-be-vetted QA pairs;
• vet each of the one or more to-be-vetted QA pairs in the to-be-vetted QA pair set through a first-pass automatic vetting procedure;
• if a vetted QA pair passes the first-pass automatic vetting procedure, add the vetted QA pair to the training set;
• and
• retrain one or more ranking models based upon the training set.