Expanding an answer key to verify a question and answer system

A method for expanding an answer key to verify a question and answer (Q and A) system, the method comprising:

• executing an answer verification application in conjunction with an artificial intelligence (AI) application, the AI application executing as a part of the Q and A system in a data processing system, the AI application configured to compute a natural language (NL) answer in response to an NL question input into the Q and A system by (i) parsing the NL question into a major feature, (ii) constructing a query from the major feature, (iii) applying the query to a corpus to select a portion of the corpus that is responsive to the NL question, (iv) executing a reasoning algorithm to perform a deep analysis of a language of the NL question and a language of the portion of the corpus, the deep analysis returning a score, the score being indicative of an extent to which a potential response is inferred by the NL question, (v) weighting the score against a statistical model to determine that the potential response is strongest in a set of potential responses, and (vi) outputting the potential response as a generated answer from the Q and A system, wherein the executing the answer verification application evaluates an accuracy of the Q and A system, the executing the answer verification application comprising:
• constructing, a definition of an extended answer type, wherein the extended answer type represents an answer type of an unrepresented answer, wherein the unrepresented answer is unrepresented in the answer key as a valid response to the NL question in a set of valid responses to the NL question in the answer key;
• creating the extended answer type in the answer key according to the definition;
• populating the extended answer type such that the unrepresented answer becomes as additional valid response to the question, the creating and the populating extending the answer key to form an extended answer key;
• and
• using the populated extended answer type in the extended answer key to verify that the generated answer from the Q and A system is correct.