Justifying passage machine learning for question and answer systems

A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system implementing a question and answer system (QA), causes the data processing system to:

• receive the input question;
• generate a set of candidate answers for the input question and, for each candidate answer in the set of candidate answers, a corresponding confidence score indicating a confidence that the corresponding candidate answer is a correct answer for the input question, and a selection of one or more selected evidence portions from a corpus of information providing evidence in support of the candidate answer being a correct answer for the input question;
• modify confidence scores of one or more of the candidate answers based on relevance scores generated by an application of a justifying passage model (JPM) to the selected evidence portions for each of the candidate answers in the set of candidate answers to thereby generate a modified set of confidence scores, wherein the JPM identifies whether a candidate answer is justified by a selected evidence passage corresponding to the candidate answer, and wherein an evidence passage justifies a corresponding candidate answer in response to content of the evidence passage explicitly stating the candidate answer to be a correct answer for the input question,
• rank the candidate answers based on the modified set of confidence scores;
• and
• output a candidate answer in the set of candidate answers as the correct answer for the input question based on the ranking of the candidate answers.

A data processing system configured to implement a question and answer system (QA), comprising:

• a processor;
• and
• a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to:
• receive the input question;
• generate a set of candidate answers for the input question and, for each candidate answer in the set of candidate answers, a corresponding confidence score indicating a confidence that the corresponding candidate answer is a correct answer for the input question, and a selection of one or more selected evidence portions from a corpus of information providing evidence in support of the candidate answer being a correct answer for the input question;
• modify confidence scores of one or more of the candidate answers based on relevance scores generated by an application of a justifying passage model (JPM) to the selected evidence portions for each of the candidate answers in the set of candidate answers to thereby generate a modified set of confidence scores, wherein the JPM identifies whether a candidate answer is justified by a selected evidence passage corresponding to the candidate answer, and wherein an evidence passage justifies a corresponding candidate answer in response to content of the evidence passage explicitly stating the candidate answer to be a correct answer for the input question;
• rank the candidate answers based on the modified set of confidence scores;
• and
• output a candidate answer in the set of candidate answers as the correct answer for the input question based on the ranking of the candidate answers.

A method, in a data processing system comprising a processor and a memory configured to implement a question and answer system (QA), for generating answers to an input question, comprising:

• receiving, in the data processing system, the input question;
• generating, by the data processing system, a set of candidate answers for the input question and, for each candidate answer in the set of candidate answers, a corresponding confidence score indicating a confidence that the corresponding candidate answer is a correct answer for the input question, and a selection of one or more selected evidence portions from a corpus of information providing evidence in support of the candidate answer being a correct answer for the input question;
• for each candidate answer in the set of candidate answers:
• applying a justifying passage model (JPM) to selected evidence portions associated with the candidate answer to generate an evidence score for each selected evidence portion, wherein the JPM identifies whether a candidate answer is justified by a selected evidence passage corresponding to the candidate answer, and wherein an evidence passage justifies a corresponding candidate answer in response to content of the evidence passage explicitly stating the candidate answer to be a correct answer for the input question;
• selecting a subset of the selected evidence portions for use in ranking the candidate answers, based on one or more filter criteria applied to the evidence scores for each selected evidence portion;
• and
• ranking the candidate answer relative to other candidate answers in the set of candidate answers based on the selected subset of the selected evidence portions;
• and
• outputting, by the data processing system, a candidate answer in the set of candidate answers as the correct answer for the input question based on the ranking of the candidate answers.