System and method of combining statistical models, data models, and human-in-the-loop for text normalization
AssigneeVerint Americas Inc.
InventorIan Beaver
Life of this patent
Estimating the term…
Abstract
According to principles described herein, unsupervised statistical models, semi-supervised data models, and HITL methods are combined to create a text normalization system that is both robust and trainable with a minimum of human intervention. This system can be applied to data from multiple sources to standardize text for insertion into knowledge bases, machine learning model training and evaluation corpora, and analysis tools and databases
Claims20
A non-transitory computer readable medium comprising instructions that, when executed by a processor of a processing system, cause the processing system to perform a method of updating a language model for a language domain, the method comprising:
- receiving text from at least two source platforms having different source text forms, the source platforms comprising interactive voice response log, social media platform, web page, or chat;
- extracting terms from the text in a source text form associated with a respective source platform to create a set of ingested terms;
- searching for the extracted terms in the source text form within a digitized data model;
- removing extracted terms that are found in the source text form in the digitized data model from the set of ingested terms;
- removing extracted terms that have a related form in the digitized data model from the set of ingested terms;
- identifying as a “new term” any term in the set of ingest terms that has not been discarded;
- assigning a priority to the new term based on context and probability of occurrence;
- and
- automatically adding the new term in the source text form to the digitized data model based on the priority above a predetermined threshold;
- and
- recompiling the language model after the new term is added to the digitized data model for a specific domain to expand vocabulary of the language model.
A method of updating a language model for a language domain, comprising:
- receiving text from at least two source platforms having different source text forms, the source platforms comprising interactive voice response log, social media platform, web page, or chat;
- extracting terms from the text in a source text form associated with a respective source platform to create a set of ingested terms;
- searching for the extracted terms in the source text form within a digitized data model;
- removing extracted terms that are found in the source text form in the digitized data model from the set of ingested terms;
- removing extracted terms that have a related form in the digitized data model from the set of ingested terms;
- identifying as a “new term” any term in the set of ingest terms that has not been discarded;
- assigning a priority to the new term based on context and probability of occurrence;
- and
- automatically adding the new term in the source text form to the digitized data model based on the priority above a predetermined threshold;
- and
- recompiling the language model after the new term is added to the digitized data model for a specific domain to expand vocabulary of the language model.
Dependencies are inferred from the claim text.