Efficient and accurate named entity recognition method and apparatus

An apparatus for recognizing and extracting named entities in a natural language input text processing comprising:

• a compressed named entities recognition (NER)-model-based named entity recognizer, configured to perform a first stage NER on the input text to generate a first stage determination of whether at least one named entity exists in the input text, wherein if the first stage NER determines at least one named entity exists in the input text, generate a first stage NER result, the first stage NER result comprises one or more pairs of named entity and its corresponding class or type;
• a rule based named entity recognizer, configured to perform a second stage NER on the input text to generate a second stage NER result if the first stage NER determines no named entity exists in the input text, wherein if the second stage NER determines at least one named entity exists in the input text, generate a second stage NER result, the second stage NER result comprises one or more pairs of named entity and its corresponding class or type;
• and
• a NER result integrator, configured to integrate the first stage NER result and the second stage NER result to generate a final NER result;
• wherein the compressed NER-model-based named entity recognizer is trained using a compressed NER model generated from a training dataset optimized by a Vector Table Optimization or a parameters optimization;
• wherein the Vector Table Optimization comprises:
• clustering of sentences in the training dataset by word vectors into a configurable number of clusters;
• selecting corpus from each of the clusters for inclusion in the training, wherein the selected corpus has not yet been selected previously for the training dataset;
• and
• omitting a portion of the training dataset in the training;
• and
• wherein the parameters optimization comprises one or more of:
• reducing word vector space dimension of the training dataset;
• reducing character vector space dimension of the training dataset;
• and
• simplifying data representation of characters using a lesser-memory-consuming data scheme in place of multi-dimensional character vectors in the training dataset.

An apparatus for recognizing and extracting named entities in a natural language input text processing comprising:

• a rule based named entity recognizer, configured to perform a named entities recognition (NER) on the input text to generate a first NER result, wherein if the first NER determines at least one named entity exists in the input text, the first NER result comprises one or more pairs of named entity and its corresponding class or type;
• wherein the rule based named entity recognizer comprises a common rules module and a specific rules module;
• wherein the common rules module is a NER classifier trained with one or more common features including top N number of most frequently used words and part-of-speech (POS) tagging of each word in a training dataset for the common rules module;
• wherein the common features are obtained from the training dataset for the common rules module by:
• splitting sentences in the training dataset for the common rules module into individual words;
• and
• extracting top N words that most frequently appear in the training dataset for the common rules module as the top N number of most frequently used words;
• and
• assigning a POS tag for each of the individual words;
• and
• wherein the specific rules module is configured to perform a second NER on the input text by regular expression identification specific to a named entity class or type and a language to generate a second NER result.