Evaluating Expert Opinions in a Question and Answer System
AssigneeInternational Business Machines Corporation
InventorsKevin B. Haverlock, Stephan J. Roorda, Michael D. Whitley
Life of this patent
Estimating the term…
Abstract
An approach is provided for evaluating subject matter experts (SMEs) in a question and answering (QA) system. In the approach, a number of responses are received at the QA system with each of the responses being a response to a common question and each of the responses being received from a SME. One of the responses is selected with the selected response being from one of the SMEs that is being evaluated. The approach evaluates the selected response by comparing the selected response to the responses received from the other SMEs. Based on the evaluation, the approach updates a SME confidence score that corresponds to the selected SME.
Claims20
A method, in an information handling system comprising a processor and a memory, of evaluating subject matter experts (SMEs) in a question and answering (QA) system, the method comprising:
- receiving, from a computer network, a plurality of responses at the QA system, wherein each of the responses pertain to a common question, where each of the responses is received from one of a plurality of SMEs, and wherein the responses are stored in the memory;
- identifying a selected one of the plurality of responses that was received from a selected one of the plurality of SMEs;
- evaluating, using the processor, the selected response by comparing the selected response to the plurality of responses;
- and
- updating, in the memory, a SME confidence score corresponding to the selected SME based on the evaluation.
An information handling system comprising:
- one or more processors;
- a memory coupled to at least one of the processors;
- a set of instructions stored in the memory and executed by at least one of the processors to evaluate subject matter experts (SMEs) in a question and answering (QA) system, wherein the set of instructions perform actions of:
- receiving a plurality of responses at the QA system, wherein each of the responses pertain to a common question, and where each of the responses is received from one of a plurality of SMEs;
- identifying a selected one of the plurality of responses that was received from a selected one of the plurality of SMEs;
- evaluating the selected response by comparing the selected response to the plurality of responses;
- and
- updating a SME confidence score corresponding to the selected SME based on the evaluation.
A computer program product stored in a computer readable storage medium, comprising computer instructions that, when executed by an information handling system, causes the information handling system to evaluate subject matter experts (SMEs) in a question and answering (QA) system by performing actions comprising:
- receiving a plurality of responses at the QA system, wherein each of the responses pertain to a common question, and where each of the responses is received from one of a plurality of SMEs;
- identifying a selected one of the plurality of responses that was received from a selected one of the plurality of SMEs;
- evaluating the selected response by comparing the selected response to the plurality of responses;
- and
- updating a SME confidence score corresponding to the selected SME based on the evaluation.
Dependencies are inferred from the claim text.