Crowdsourcing to filter out unpopular potential candidate answers
AssigneeInternational Business Machines Corporation
InventorsHuaiyu Zhu, Yunyao Li, Youxuan Jiang
Life of this patent
Estimating the term…
Abstract
Technology for selecting a correct answer (for example, a correct label for a data set to be used in machine learning algorithms) from among a plurality of candidate answers, where the answers selected relatively infrequently by a plurality of human evaluators are cold from the full plurality of candidate answers to obtain a reduced subset of candidate answers. In this way, further selection of the correct answer (for example, ultimate selection of the correct answer by a human expert) will only need to consider the reduced subset, thereby potentially saving time and effort in the selection of the correct answer.
Claims20
A method for selecting among a plurality of candidate answers by a plurality of human nonexpert annotators and at least one expert annotator, the method comprising:
- filtering out one or more rejected candidate answers from the plurality of candidate answers based upon the one or more rejected candidate answers being chosen relatively infrequently as a correct answer by the plurality of human nonexpert annotators to obtain a reduced subset of candidate answers;
- and
- selecting, from the reduced subset of candidate answers, an optimal answer based upon responses received from at least one of the following:
- the plurality of human nonexpert annotators and the at least one human expert annotator.
A computer implemented method (CIM) comprising:
- receiving a training data set designed for use in a machine learning computer system;
- receiving a plurality of candidate labels for the training data set;
- receiving, respectively from a plurality of human individual(s), a plurality of responses with each response including information indicative of one, or more, candidate label(s) of the plurality of candidate labels that a human individual providing the response considers to be most appropriate to describe content of the training data set;
- determining, by machine logic, a rejected subset of rejected candidate label(s), the rejected subset including at least one candidate label of the plurality of candidate labels, with the determination being based upon each rejected candidate label being chosen in the plurality of responses with a frequency below a threshold frequency;
- and
- determining, by machine logic, an accepted subset of accepted candidate label(s), the accepted subset including at least one candidate label of the plurality of candidate labels, with the determination being made by taking as accepted candidate label(s) all of the potential candidate labels that are not rejected candidate labels.
A computer implemented method (CIM) comprising:
- receiving a training data set designed for use in a machine learning computer system, with the training data set including a first label;
- receiving a plurality of candidate labels for the training data set, with one candidate label of the plurality of candidate labels being the first data set;
- receiving, respectively from a plurality of human individual(s), a plurality of responses with each response including information indicative of one, or more, candidate label(s) of the plurality of candidate labels that a human individual providing the response considers to be most appropriate to describe content of the training data set;
- determining, by machine logic, that the first label was chosen in the plurality of responses with a frequency below a threshold frequency;
- responsive to the determination that the first label was chosen with a frequency below the threshold frequency, selecting a second label of the plurality of candidate labels as being most appropriate to describe the content of the training data set;
- and
- replacing, in the training data set, the first label with the second label.
Dependencies are inferred from the claim text.