System and method for unsupervised text normalization using distributed representation of words
AssigneeAT&T Intellectual Property I, L.P.
InventorVivek Kumar RANGARAJAN SRIDHAR
Life of this patent
Estimating the term…
Abstract
A system, method and computer-readable storage devices for providing unsupervised normalization of noisy text using distributed representation of words. The system receives, from a social media forum, a word having a non-canonical spelling in a first language. The system determines a context of the word in the social media forum, identifies the word in a vector space model, and selects an “n-best” vector paths in the vector space model, where the n-best vector paths are neighbors to the vector space path based on the context and the non-canonical spelling. The system can then select, based on a similarity cost, a best path from the n-best vector paths and identify a word associated with the best path as the canonical version.
Claims20
A method comprising:
- receiving, from a social media forum, a word having a non-canonical spelling;
- identifying the word in a vector space model, to yield a vector space path, wherein the vector space model is trained from a corpus of noisy text, and wherein words within the vector space model are clustered based on context;
- selecting n-best vector paths in the vector space model which are neighbors to the vector space path on the non-canonical spelling;
- selecting, based on a similarity cost, a best path from the n-best vector paths;
- and
- nominating a word associated with the best path as a canonical form.
A system comprising:
- a processor;
- and
- a computer-readable storage medium having instructions stored which, when executed by the processor, result in operations comprising:
- receiving, from a social media forum, a word having a non-canonical spelling in a first language;
- identifying the word in a vector space model, to yield a vector space path, wherein the vector space model is trained from a corpus of noisy text, and wherein words within the vector space model are clustered based on context;
- selecting n-best vector paths in the vector space model which are neighbors to the vector space path on the non-canonical spelling;
- selecting, based on a similarity cost, a best path from the n-best vector paths;
- and
- nominating a word associated with the best path as a canonical form.
A computer-readable storage device having instructions stored which, when executed by a computing device, result in operations comprising:
- receiving, from a social media forum, a word having a non-canonical spelling in a first language;
- identifying the word in a vector space model, to yield a vector space path, wherein the vector space model is trained from a corpus of noisy text, and wherein words within the vector space model are clustered based on context;
- selecting n-best vector paths in the vector space model which are neighbors to the vector space path on the non-canonical spelling;
- selecting, based on a similarity cost, a best path from the n-best vector paths;
- and
- nominating a word associated with the best path as a canonical form.
Dependencies are inferred from the claim text.