US 11,232,255 B2 · Generating digital annotations for evaluating and training automatic electronic document annotation models

Abstract

Systems, methods, and non-transitory computer-readable media are disclosed that collect and analyze annotation performance data to generate digital annotations for evaluating and training automatic electronic document annotation models. In particular, in one or more embodiments, the disclosed systems provide electronic documents to annotators based on annotator topic preferences. The disclosed systems then identify digital annotations and annotation performance data such as a time period spent by an annotator in generating digital annotations and annotator responses to digital annotation questions. Furthermore, in one or more embodiments, the disclosed systems utilize the identified digital annotations and the annotation performance data to generate a final set of reliable digital annotations. Additionally, in one or more embodiments, the disclosed systems provide the final set of digital annotations for utilization in training a machine learning model to generate annotations for electronic documents.

Claims20

Dependent claims

In a digital medium environment for evaluating and training electronic document annotation models, a method of generating accurate ground-truth digital annotations for digital documents comprising:

identifying digital annotations generated by client devices of a plurality of annotators for an electronic document;
determining annotation performance data for the plurality of annotators utilizing annotator interactions corresponding to the electronic document via the client devices of the plurality of annotators, the annotator interactions comprising explored percentages of the electronic document and unexplored percentages of the electronic document for the plurality of annotators determined from scrolling actions within the electronic document;
generating a final set of digital annotations for the electronic document by filtering the digital annotations based on a comparison between the annotation performance data comprising the explored percentages and the unexplored percentages for the plurality of annotators in relation to the electronic document and one or more threshold performance metrics;
and
providing the final set of digital annotations as ground truth data associated with the electronic document for utilization in at least one of:
testing an annotation model or training the annotation model to automatically generate document annotations.

3 more elements

A system comprising:

at least one processor;
and
at least one non-transitory computer-readable storage medium storing instructions thereon that, when executed by the at least one processor, cause the system to:
identify a plurality of digital annotations, the plurality of digital annotations comprising one or more digital annotations generated for an electronic document via a client device of an annotator;
provide one or more digital annotation questions corresponding to the electronic document to the client device of the annotator;
determine annotation performance data based on an explored percentage of the electronic document and an unexplored percentage of the electronic document determined from scrolling actions within the electronic document via the client device annotator responses to the one or more digital annotation questions;
generate a final set of digital annotations for the electronic document by filtering the one or more digital annotations based on a comparison between the annotation performance data comprising the explored percentage and the unexplored percentage for the annotator in relation to the electronic document and the annotator responses and one or more threshold performance metrics;
and
provide the final set of digital annotations as ground truth data associated with the electronic document for utilization in at least one of:
testing an annotation model or training a machine learning model to generate annotations for input electronic documents.

7 more elements

In a digital medium environment for evaluating and training electronic document annotation models, a non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause a computer device to:

identify digital annotations generated by client devices of a plurality of annotators for an electronic document;
determine annotation performance data for the plurality of annotators utilizing annotator interactions corresponding to the electronic document via the client devices of the plurality of annotators, the annotator interactions comprising explored percentages of the electronic document and unexplored percentages of the electronic document for the plurality of annotators determined from scrolling actions within the electronic document;
generate a final set of digital annotations for the electronic document by filtering the digital annotations based on a comparison between the annotation performance data comprising the explored percentages and the unexplored percentages for the plurality of annotators in relation to the electronic document and one or more threshold performance metrics;
and
provide the final set of digital annotations as ground truth data associated with the electronic document for utilization in at least one of:
testing an annotation model or training the annotation model to automatically generate digital document annotations.

3 more elements

Dependencies are inferred from the claim text.

Generating digital annotations for evaluating and training automatic electronic document annotation models

Life of this patent

Generating digital annotations for evaluating and training automatic electronic document annotation models

Life of this patent

Abstract

Claims20