Dynamic document reliability formulation
AssigneeInternational Business Machines Corporation
InventorsKeith G. Frost, Stephen A. Boxwell, Stanley J. Vernier, Kyle M. Brake
Life of this patent
Estimating the term…
Abstract
Embodiments relate to an intelligent computer platform to use ground truth data to rate source reliability, and application of the source reliability. A computer system, computer program product, and computer implemented method apply the ground truth data to evaluate a document and statements therein, and dynamically apply the assessed reliability. The assessed reliability is dynamically applied to a source, with the application yielding acceptance of the document to the source based on the applied reliability assessment.
Claims20
A computer system comprising:
- a processing unit operatively coupled to memory;
- and
- a knowledge engine in communication with the processing unit, the knowledge engine configured to evaluate document reliability, the knowledge engine including:
- a natural language processing (NLP) manager configured to identify statements in one or more documents;
- an assessment manager operatively coupled to the NLP manager and configured to:
- apply ground truth question and answer data to the statements to generate response data from the statements, the response data comprising content of the one or more documents, wherein the ground truth question and answer data are vetted for factual accuracy prior to application to the statements;
- and
- evaluate accuracy of the statements based on the ground truth question and answer data and the generated response data;
- and
- a director, operatively coupled to the assessment manager, the director configured to apply the evaluated accuracy of the statements to the one or more identified documents, including to:
- calculate document reliability of the one or more documents based on the evaluated accuracy across each of the evaluated statements;
- and
- the assessment manager configured to selectively apply the one or more documents to a source based on the calculated document reliability.
A computer program product for dynamically assessing reliability of one or more documents, the computer program product comprising:
- a computer readable storage medium;
- and
- program code embodied with the computer readable storage medium, the program code executable by a processor to:
- leverage natural language processing (NLP) to identify statements in one or more documents;
- apply ground truth question and answer data to the statements to generate response data from the statements, the response data comprising content of the one or more documents, wherein the ground truth question and answer data are vetted for factual accuracy prior to application to the statements;
- and
- evaluate accuracy of the statements based on the ground truth question and answer data and the generated response data;
- and
- apply the evaluated accuracy of the statements to the one or more documents, including calculate document reliability of the one or more documents based on the evaluated accuracy across each of the evaluated statements;
- and
- selectively apply the one or more documents to a source based on the calculated document reliability.
A computer-implemented method for assessing reliability of one or more documents by an artificial intelligence platform, including:
- using natural language processing (NLP), identifying statements in one or more documents;
- applying ground truth question and answer data to the statements to generate response data from the statements, the response data comprising content of the one or more documents, wherein the ground truth question and answer data are vetted for factual accuracy prior to application to the statements;
- evaluating accuracy of the statements based on the ground truth question and answer data and the generated response data;
- applying the evaluated accuracy of the statements to the one or more identified documents, including calculating document reliability of the one or more documents based on the evaluated accuracy across each of the evaluated statements;
- and
- selectively applying the one or more documents to a source based on the calculated document reliability.
Dependencies are inferred from the claim text.