Annotation alignment algorithm analysis for character recognition

A method performed by a computer, the method comprising:

• receiving, from a first source including an annotation alignment service or associated storage device, weak-to-strong annotations for a document that are generated by applying a weak-to-strong annotation alignment algorithm that has weak annotations for the document as an input;
• receiving, from a second source, strong annotations for the document;
• matching strong annotations from the second source to weak-to-strong annotations from the first source to generate respective pairs of matched annotations;
• for each pair of matched annotations, calculating respective metrics for the strong annotation of the pair and the weak-to-strong annotation of the pair, the respective metrics including one or more of:
• an exact match metric comparing Optical Character Recognition (OCR) text in a first bounding box of the weak-to-strong annotation of the pair to a string of the strong annotation of the pair, an exact raw match metric comparing the OCR text in the first bounding box to OCR text in a second bounding box of the strong annotation of the pair, or an intersection over union metric corresponding to an overlap of the first and second bounding boxes;
• aggregating the respective metrics for the pairs of matched annotations to generate aggregated metrics;
• and
• outputting an indication of the aggregated metrics to a graphical user interface or a target application, the indication usable to determine a selection or adjustment of the weak-to-strong annotation alignment algorithm for use in generating subsequent strong annotations from weakly annotated data sets for use in OCR operations and to cause the determined selection or adjustment of the weak-to-strong annotation alignment algorithm to be used in generating subsequent strong annotations.

A system, comprising:

• one or more hardware processors with memory coupled thereto;
• computer-readable media storing instructions executable by the one or more hardware processors, the instructions comprising:
• first instructions to generate an expanded annotated data set for a document, the expanded annotated data set including strong annotations of optical character recognition (OCR) labels for content in the document, each strong annotation including a respective first bounding box and a string, and weak-to-strong annotations resulting from applying a first weak-to-strong annotation alignment algorithm to the labels, each weak-to-strong annotation including a respective second bounding box generated by the weak-to-strong annotation alignment algorithm;
• second instructions to, for each label of the OCR labels, match a respective strong annotation to a respective weak-to-strong annotation for the label to form a respective matched pair;
• third instructions to calculate individual metrics for the strong annotations and for the weak-to-strong annotations in the expanded annotated data set, the individual metrics including different types of comparisons of aspects of the strong annotations to the weak-to-strong annotations for each respective matched pair;
• fourth instructions to aggregate each of the different types of the individual metrics associated with each label of the OCR labels to generate first aggregated metrics;
• fifth instructions to execute the first, second, third, and fourth instructions again using a second weak-to-strong annotation alignment algorithm to generate second aggregated metrics;
• sixth instructions to compare the first aggregated metrics to the second aggregated metrics;
• and
• seventh instructions to output an indication of the comparison of the first aggregated metrics to the second aggregated metrics to a graphical user interface or a target application, the indication usable to cause a selection or adjustment of the first or second weak-to-strong annotation alignment algorithm for use in generating subsequent strong annotations from weakly annotated data sets for use in Optical Character Recognition (OCR) operations and to cause the selection or adjustment of the first or second weak-to-strong annotation alignment algorithm to be used in generating subsequent strong annotations.