Exploiting answer key modification history for training a question and answering system
AssigneeInternational Business Machines Corporation
InventorsEdward G. Katz, John A. Riendeau, Sean T. Thatcher, Alexander C. Tonetti
Life of this patent
Estimating the term…
Abstract
A system and a computer program product are provided for evaluating question-answer pairs in an answer key by generating a predicted answer to a test question based on the answer key modification history for comparison matching against a generated answer that is generated in response to the test question, and then comparing the predicted answer and generated answer to determine an accuracy score match indication therebetween so as to present an indication that the answer key may have a problem if there is a match between the predicted answer and generated answer.
Claims12
An information handling system comprising:
- one or more processors;
- a memory coupled to at least one of the processors;
- a set of instructions stored in the memory and executed by at least one of the processors to evaluate an answer key for accuracy using an answer key modification history, wherein the set of instructions are executable to perform actions of:
- receiving, by the system, a first answer generated from a system capable of answering questions in response to a test question from an answer key;
- generating, by the system, a predicted answer to the test question based on an answer key modification history associated with the test question if there is not a match between the first answer and the answer key;
- training, by the system, a machine learning process to evaluate, the predicted answer against the first answer to identify an answer key update candidate from the predicted answer or the first answer that improves a functional accuracy of the answer key;
- generating, by the system, an accuracy score based on how closely the predicted answer matches the first answer;
- and
- presenting, by the system, an indication that the answer key should be updated with the answer key update candidate if the accuracy score indicates there is a match between the predicted answer and first answer.
A computer program product stored in a computer readable storage medium, comprising computer instructions that, when executed by an information handling system, causes the system to evaluate question-answer pairs in an answer key by:
- receiving, by the system, a first answer generated from a system capable of answering questions in response to a test question from an answer key;
- generating, by the system, a predicted answer to the test question based on an answer key modification history associated with the test question if there is not a match between the first answer and the answer key;
- training, by the system, a machine learning process to evaluate, the predicted answer against the first answer to identify an answer key update candidate from the predicted answer or the first answer that improves a functional accuracy of the answer key;
- generating, by the system, an accuracy score based on how closely the predicted answer matches the first answer;
- and
- presenting, by the system, an indication that the answer key should be updated with the answer key update candidate predicted answer or the first answer if the accuracy score indicates there is a match between the predicted answer and first answer.
Dependencies are inferred from the claim text.