# US 12,118,303 B2 · Method and system for text cleaning > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,303 B2 - Document ID: US12118303B2 - Status: Granted - Kind: B2 - Application number: 17/474,694 - Filed: 2021-09-14 - Published: 2022-06-02 - Granted: 2024-10-15 - Earliest claimed: 2020-11-30 - Assignees: Tata Consultancy Services Limited - Applicants: Tata Consultancy Services Limited - Inventors: Gargi Roy; Lipika Dey; Mohammad Shakir - Web page: https://patentgrep.com/patent/US12118303B2 ## CPC - G06F 40/232: Orthographic correction, e.g. spell checking or vowelisation - G06F 40/242: Dictionaries - G06F 18/22: Matching criteria, e.g. proximity measures - G06F 18/2431: Multiple classes - G06F 40/166: Editing, e.g. inserting or deleting - G06F 40/279: Recognition of textual entities - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract State of the art systems used for text processing fail to differentiate between domain terms and noise terms that form a major part of enterprise and social media data. The disclosure herein generally relates to text processing, and, more particularly, to segregating domain words and noise terms from the text. The system processes input data and classifies words as dictionary words and non-dictionary words. The system processes the dictionary words and non-dictionary words and identifies domain terms and noise terms. The system then corrects the words identified as the noise terms. ## Claims ### Claim 1 A processor implemented method of text cleaning, comprising: collecting data comprising a plurality of words as input data, via one or more hardware processors; classifying the plurality of words to a set of dictionary words (V) and a set of non-dictionary words (N), via the one or more hardware processors; determining a word frequency of each word in V and N, via the one or more hardware processors, wherein the word frequency of a word represents a frequency of an occurrence of the word in the input data; segregating the non-dictionary words in N as domain terms and noise terms, based on the determined word frequency, via the one or more hardware processors; classifying the plurality of words in V and N into a plurality of pre-defined categories, via the one or more hardware processors, wherein the plurality of pre-defined categories comprises a plurality of noise categories, a plurality of error categories and a plurality of non-error categories; determining correct term for each of a plurality of noise terms in the plurality of noise categories among the plurality of pre-defined categories, via the one or more hardware processors, wherein determining the correct term for each of the plurality of noise terms comprises: removing stop words from the input data to generate pre-processed data; determining context of the plurality of words by generating word embeddings of the plurality of words; generating character embedding of the plurality of words, wherein the character embedding of new words in the pre-processed data have at least partial overlap with known words with same root form in the pre-processed data; determining edit distance between the plurality of words; determining phoneme based distance between the plurality of words; and determining the correct term of each of the plurality of noise terms in N as a function of cosine similarity of the word embeddings, the character embedding, the determined edit distance, and the determined phoneme based distance; reclassifying the corrected terms, via the one or more hardware processors, wherein the reclassification of the corrected terms involves movement of the corrected terms from one category to another category among the plurality of pre-defined categories; and replacing the noise terms in the input data with the corresponding corrected terms, via the one or more hardware processors. ### Claim 2 (depends on 1) The processor implemented method as claimed in claim 1, wherein classifying the plurality of words in V and N into the plurality of pre-defined categories, comprises: classifying all of the plurality of words in V which fall in Jenk's class 1 as dictionary domain terms belonging to a first category of the plurality of categories; classifying all of the plurality of words in N which fall in Jenk's class 1 as non-dictionary domain terms belonging to a second category of the plurality of categories; determining each noise word identified as a distortion of a dictionary term as belonging to a third category of the plurality of categories; and determining each noise word identified as a distortion of a non-dictionary term as belonging to a fourth category of the plurality of categories. ### Claim 3 (depends on 2) The processor implemented method as claimed in claim 2, wherein reclassifying the corrected terms comprises iteratively performing the classification of the plurality of words into the plurality of categories and the movement of the corrected terms from one category to another category till no further movement of the corrected terms between the pre-defined categories is possible. ### Claim 4 A system for document correction, comprising: a memory storing instructions; one or more communication interfaces; and one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to: collect data comprising a plurality of words as input data; classify the plurality of words to a set of dictionary words (V) and a set of non-dictionary words (N); determine a word frequency of each word in V and N, wherein the word frequency of a word represents a frequency of a occurrence of the word in the input data; segregate the non-dictionary words in N as domain terms and noise terms, based on the determined word frequency; classify the plurality of words in V and N into a plurality of pre-defined categories, wherein the plurality of pre-defined categories comprises a plurality of noise categories, a plurality of error categories and a plurality of non-error categories; determine correct term for each of a plurality of noise terms in the plurality of noise categories among the plurality of pre-defined categories, wherein the system determines the correct term for each of the plurality of noise terms by: removing stop words from the input data to generate pre-processed data; determining context of the plurality of words by generating word embeddings of the plurality of words; generating character embedding of the plurality of words, wherein the character embedding of new words in the pre-processed data have at least partial overlap with known words with same root form in the pre-processed data; determining edit distance between the plurality of words; determining phoneme based distance between the plurality of words; and determining the correct term of each of the plurality of noise terms in N as a function of cosine similarity of the word embeddings, the character embedding, the determined edit distance, and the determined phoneme based distance; reclassify the corrected terms, wherein the reclassification of the corrected terms involves movement of the corrected terms from one category to another category among the plurality of pre-defined categories; and replace the noise terms in the input data with the corresponding corrected terms. ### Claim 5 (depends on 4) The system as claimed in claim 4, wherein the system classifies the plurality of words in V and N into the plurality of pre-defined categories by: classifying all of the plurality of words in V which fall in Jenk's class 1 as dictionary domain terms belonging to a first category of the plurality of categories; classifying all of the plurality of words in N which fall in Jenk's class 1 as non-dictionary domain terms belonging to a second category of the plurality of categories; determining each noise word identified as a distortion of a dictionary term as belonging to a third category of the plurality of categories; and determining each noise word identified as a distortion of a non-dictionary term as belonging to a fourth category of the plurality of categories. ### Claim 6 (depends on 5) The system as claimed in claim 5, wherein the system reclassifies the corrected terms by iteratively performing the classification of the plurality of words into the plurality of categories and the movement of the corrected terms from one category to another category till no further movement of the corrected terms between the pre-defined categories is possible. ### Claim 7 A non-transitory computer readable medium for document correction, wherein the non-transitory computer readable medium comprising a plurality of instructions, which when executed, cause: collecting data comprising a plurality of words as input data, via one or more hardware processors; classifying the plurality of words to a set of dictionary words (V) and a set of non-dictionary words (N), via the one or more hardware processors; determining a word frequency of each word in V and N, via the one or more hardware processors, wherein the word frequency of a word represents a frequency of an occurrence of the word in the input data; segregating the non-dictionary words in N as domain terms and noise terms, based on the determined word frequency, via the one or more hardware processors; classifying the plurality of words in V and N into a plurality of pre-defined categories, via the one or more hardware processors, wherein the plurality of pre-defined categories comprises a plurality of noise categories, a plurality of error categories and a plurality of non-error categories; determining correct term for each of a plurality of noise terms in the plurality of noise categories among the plurality of pre-defined categories, via the one or more hardware processors, wherein determining the correct term for each of the plurality of noise terms comprises: removing stop words from the input data to generate pre-processed data; determining context of the plurality of words by generating word embeddings of the plurality of words; generating character embedding of the plurality of words, wherein the character embedding of new words in the pre-processed data have at least partial overlap with known words with same root form in the pre-processed data; determining edit distance between the plurality of words; determining phoneme based distance between the plurality of words; and determining the correct term of each of the plurality of noise terms in N as a function of cosine similarity of the word embeddings, the character embedding, the determined edit distance, and the determined phoneme based distance; reclassifying the corrected terms, via the one or more hardware processors, wherein the reclassification of the corrected terms involves movement of the corrected terms from one category to another category among the plurality of pre-defined categories; and replacing the noise terms in the input data with the corresponding corrected terms, via the one or more hardware processors. ### Claim 8 (depends on 7) The non-transitory computer readable medium as claimed in claim 7, wherein the non-transitory computer readable medium causes classification of the plurality of words in V and N into the plurality of pre-defined categories, by: classifying all of the plurality of words in V which fall in Jenk's class 1 as dictionary domain terms belonging to a first category of the plurality of categories; classifying all of the plurality of words in N which fall in Jenk's class 1 as non-dictionary domain terms belonging to a second category of the plurality of categories; determining each noise word identified as a distortion of a dictionary term as belonging to a third category of the plurality of categories; and determining each noise word identified as a distortion of a non-dictionary term as belonging to a fourth category of the plurality of categories. ### Claim 9 (depends on 8) The non-transitory computer readable medium as claimed in claim 8, wherein the non-transitory computer readable medium causes reclassification of the corrected terms by iteratively performing the classification of the plurality of words into the plurality of categories and the movement of the corrected terms from one category to another category till no further movement of the corrected terms between the pre-defined categories is possible.