US 2018/0322411 A1Published application

Automatic evaluation and validation of text mining algorithms

AssigneeLinkedIn Corporation

InventorsHu Wang, Chi-Yi Kuan, Yongzheng Zhang, Rui Zhao

Life of this patent

Estimating the term…

Loading patent…

Abstract

In some embodiments, the disclosed subject matter involves comparing the results of natural language processing (NLP) of unstructured text to historical results for verification and validation of the NLP models/algorithms. The analysis uses statistical theory and practices to automatically monitor and validate the performances of the (NLP) algorithms on a periodic basis. Each unstructured text is run through one or more NLP algorithms and scored for relevance or contextual classification. Distribution of the scores is assumed to be Gaussian in nature so that a probability value (p-value) may be generated. When the p-value is below a threshold value, manual tagging may be initiated for the current time period to help retrain the models for better performance. Other embodiments are described and claimed.

Claims20

Dependent claims

A confidence validation system, comprising:

a processor coupled to a storage medium including instructions stored thereon, the instructions when executed cause a machine to:
receive a plurality of unstructured text items for a current time period;
analyze each of the plurality of unstructured text items for relevance or contextual classification to tag each of the plurality of unstructured text items with identified relevance or contextual classification, the analyzing to use at least one logic module for natural language processing;
generate at least one tagged data set based at least on the analyzing of the plurality unstructured text items;
store the at least one tagged data set in an historic database communicatively coupled to the processor;
perform automatic analysis of a first tagged data set for the current time period as compared to historical tagged data sets for m number of time periods, the instructions for automatic analysis to include instructions to identify a statistical p-value for the first tagged data set for the current time period as compared to a Gaussian distribution of the m time periods of historical tagged data sets;
and
determine whether the at least one tagged data set for the current time period falls outside of expected results.

5 more elements

A computer implemented method, comprising:

receiving a plurality of unstructured text items for a current time period;
analyzing each of the plurality of unstructured text items for relevance or contextual classification to tag each of the plurality of unstructured text items with identified relevance or contextual classification, the analyzing to use at least one logic module for natural language processing;
generating at least one tagged data set based at least on the analyzing of the plurality unstructured text items;
storing the at least one tagged data set in an historic database;
performing automatic analysis of a first tagged data set for the current time period as compared to historical tagged data sets for m number of time periods;
identifying a statistical p-value for the first tagged data set for the current time period as compared to a Gaussian distribution of the m time periods of historical tagged data sets;
and
determining whether the at least one tagged data set for the current time period falls outside of expected results.

5 more elements

Dependencies are inferred from the claim text.