Generating multiple choice questions and answers based on document text

An information handling system comprising:

• one or more processors;
• one or more data stores accessible by at least one of the processors;
• a memory coupled to at least one of the processors;
• and
• a set of computer program instructions stored in the memory and executed by at least one of the processors in order to improve a question answering (QA) computer system by performing actions of:
• ingesting a set of documents into a first corpus of a first QA computer system;
• generating a multiple choice test from the set of documents ingested into the first corpus, wherein the multiple choice test includes a plurality of questions, with each of the questions having a set of candidate answers that includes one correct answer and one or more incorrect candidate answers, wherein the questions and answers are retrieved from the set of documents;
• identifying one or more concepts corresponding to each of the questions;
• generating the incorrect candidate answers corresponding to each of the questions based on the identified concepts pertaining to the respective questions, wherein the incorrect candidate answers are designed to be misleading in light of the respective questions;
• and
• testing a subject using the generated multiple choice test, wherein the subject is a second QA computer system that utilizes a second corpus, with the testing further comprising:
• submitting a selected one of the generated questions from the multiple choice test to the second QA computer system;
• receiving a responsive answer from the second QA computer system;
• comparing the responsive answer to the correct answer and candidate answers corresponding to the selected generated question;
• and
• improving an accuracy of the second QA computer system by updating the second corpus with the correct answer to the selected question in response to the comparison revealing that the second QA computer system selected one of the incorrect candidate answers.

A computer program product stored in a computer readable storage medium device, comprising computer program code that, when executed by an information handling system, causes the information handling system to improve a question answering (QA) computer system by performing actions comprising:

• ingesting a set of documents into a first corpus of a first QA computer system;
• generating a multiple choice test from the set of documents ingested into the first corpus, wherein the multiple choice test includes a plurality of questions, with each of the questions having a set of candidate answers that includes one correct answer and one or more incorrect candidate answers, wherein the questions and answers are retrieved from the set of documents;
• identifying one or more concepts corresponding to each of the questions;
• generating the incorrect candidate answers corresponding to each of the questions based on the identified concepts pertaining to the respective questions, wherein the incorrect candidate answers are designed to be misleading in light of the respective questions;
• and
• testing a subject using the generated multiple choice test, wherein the subject is a second QA computer system that utilizes a second corpus, with the testing further comprising:
• submitting a selected one of the generated questions from the multiple choice test to the second QA computer system;
• receiving a responsive answer from the second QA computer system;
• comparing the responsive answer to the correct answer and candidate answers corresponding to the selected generated question;
• and
• improving an accuracy of the second QA computer system by updating the second corpus with the correct answer to the selected question in response to the comparison revealing that the second QA computer system selected one of the incorrect candidate answers.