Evaluating quality of annotation

A computer program product for evaluating annotation quality, the computer program product comprising at least one computer readable non-transitory storage medium having computer readable program instructions thereon for execution by a processor, the computer readable program instructions comprising program instructions for:

• obtaining annotation information associated with a plurality of annotators and a plurality of data elements, wherein the annotation information comprises a plurality of annotation entries corresponding to at least one data element from the plurality of data elements, and wherein the plurality of annotation entries are entered based on an annotation guideline;
• determining a quality rating for the annotation guideline, wherein the determining is based on a comparison between a first value associated with the plurality of annotators and the plurality of data elements and a second value associated with any disparity among the plurality of annotation entries;
• determining a proficiency rating for an annotator from the plurality of annotators, wherein the determining is based on a comparison between a third value associated with annotation entries by the annotator and the second value associated with any disparity among the plurality of annotation entries;
• and
• generating a report based on the quality rating and the proficiency rating.

A computer system for evaluating annotation quality, the computer system comprising:

• at least one processing unit;
• at least one computer readable memory;
• at least one computer readable tangible, non-transitory storage medium;
• and program instructions stored on the at least one computer readable tangible, non-transitory storage medium for execution by the at least one processing unit via the at least one computer readable memory, wherein the program instructions comprise program instructions for:
• obtaining annotation information associated with a plurality of annotators and a plurality of data elements, wherein the annotation information comprises a plurality of annotation entries corresponding to at least one data element from the plurality of data elements, and wherein the plurality of annotation entries are entered based on an annotation guideline;
• determining a quality rating for the annotation guideline, wherein the determining is based on a comparison between a first value associated with the plurality of annotators and the plurality of data elements and a second value associated with any disparity among the plurality of annotation entries;
• determining a proficiency rating for an annotator from the plurality of annotators, wherein the determining is based on a comparison between a third value associated with annotation entries by the annotator and the second value associated with any disparity among the plurality of annotation entries;
• and
• generating a report based on the quality rating and the proficiency rating.