Method and system for text cleaning

A system for document correction, comprising:

• a memory storing instructions;
• one or more communication interfaces;
• and
• one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to:
• collect data comprising a plurality of words as input data;
• classify the plurality of words to a set of dictionary words (V) and a set of non-dictionary words (N);
• determine a word frequency of each word in V and N, wherein the word frequency of a word represents a frequency of a occurrence of the word in the input data;
• segregate the non-dictionary words in N as domain terms and noise terms, based on the determined word frequency;
• classify the plurality of words in V and N into a plurality of pre-defined categories, wherein the plurality of pre-defined categories comprises a plurality of noise categories, a plurality of error categories and a plurality of non-error categories;
• determine correct term for each of a plurality of noise terms in the plurality of noise categories among the plurality of pre-defined categories,
• wherein the system determines the correct term for each of the plurality of noise terms by:
• removing stop words from the input data to generate pre-processed data;
• determining context of the plurality of words by generating word embeddings of the plurality of words;
• generating character embedding of the plurality of words, wherein the character embedding of new words in the pre-processed data have at least partial overlap with known words with same root form in the pre-processed data;
• determining edit distance between the plurality of words;
• determining phoneme based distance between the plurality of words;
• and
• determining the correct term of each of the plurality of noise terms in N as a function of cosine similarity of the word embeddings, the character embedding, the determined edit distance, and the determined phoneme based distance;
• reclassify the corrected terms, wherein the reclassification of the corrected terms involves movement of the corrected terms from one category to another category among the plurality of pre-defined categories;
• and
• replace the noise terms in the input data with the corresponding corrected terms.

A non-transitory computer readable medium for document correction, wherein the non-transitory computer readable medium comprising a plurality of instructions, which when executed, cause:

• collecting data comprising a plurality of words as input data, via one or more hardware processors;
• classifying the plurality of words to a set of dictionary words (V) and a set of non-dictionary words (N), via the one or more hardware processors;
• determining a word frequency of each word in V and N, via the one or more hardware processors, wherein the word frequency of a word represents a frequency of an occurrence of the word in the input data;
• segregating the non-dictionary words in N as domain terms and noise terms, based on the determined word frequency, via the one or more hardware processors;
• classifying the plurality of words in V and N into a plurality of pre-defined categories, via the one or more hardware processors, wherein the plurality of pre-defined categories comprises a plurality of noise categories, a plurality of error categories and a plurality of non-error categories;
• determining correct term for each of a plurality of noise terms in the plurality of noise categories among the plurality of pre-defined categories, via the one or more hardware processors, wherein determining the correct term for each of the plurality of noise terms comprises:
• removing stop words from the input data to generate pre-processed data;
• determining context of the plurality of words by generating word embeddings of the plurality of words;
• generating character embedding of the plurality of words, wherein the character embedding of new words in the pre-processed data have at least partial overlap with known words with same root form in the pre-processed data;
• determining edit distance between the plurality of words;
• determining phoneme based distance between the plurality of words;
• and
• determining the correct term of each of the plurality of noise terms in N as a function of cosine similarity of the word embeddings, the character embedding, the determined edit distance, and the determined phoneme based distance;
• reclassifying the corrected terms, via the one or more hardware processors, wherein the reclassification of the corrected terms involves movement of the corrected terms from one category to another category among the plurality of pre-defined categories;
• and
• replacing the noise terms in the input data with the corresponding corrected terms, via the one or more hardware processors.