Systems and methods of enhanced clustering based on word-bound n-grams
AssigneeMastercard International Incorporated
InventorTadeu Augusto FERREIRA
Life of this patent
Estimating the term…
Abstract
A system may access a dataframe having a plurality of records, each record having a respective string of characters. The system may, for each record, from among the plurality of records: generate a plurality of word-bound N-grams based on the respective string of characters of the record, wherein N is an integer number, generate a feature vector for the record based on the plurality of word-bound N-grams. The system may determine levels of similarity between the plurality of records based on the generated feature vectors. The system may cluster the plurality of records into clusters of records based on the levels of similarity. The system may restructure the dataframe to generate a restructured dataframe based on the clusters.
Claims20
A system, comprising:
- a processor programmed to:
- access a dataframe having a plurality of records, each record having a respective string of characters;
- for each record, from among the plurality of records:
- identify individual words from the respective string in the record;
- generate a plurality of word-bound N-grams based on the respective string of characters of the record, wherein N is an integer number and wherein a last word-bound N-gram for each word includes the last N characters of the word or less than the last N characters of the word;
- generate a feature vector for the record based on the plurality of word-bound N-grams;
- determine levels of similarity between the plurality of records based on the feature vectors;
- cluster the plurality of records into clusters of records based on the levels of similarity;
- and
- restructure the dataframe into a restructured dataframe based on the clusters of records.
A method, comprising:
- accessing, by a processor, a dataframe having a plurality of records, each record having a respective string of characters;
- for each record, from among the plurality of records:
- generating, by the processor, a plurality of word-bound N-grams based on the respective string of characters of the record, wherein N is an integer number;
- identifying, by the processor, individual words from the respective string in the record;
- generating, by the processor, a feature vector for the record based on the plurality of word-bound N-grams and wherein a last word-bound N-gram for each word includes the last N characters of the word or less than the last N characters of the word;
- determining, by the processor, levels of similarity between the plurality of records based on the feature vectors;
- clustering, by the processor, the plurality of records into clusters of records based on the levels of similarity;
- and
- restructuring, by the processor, the dataframe into a restructured dataframe based on the clusters of records.
A non-transitory computer readable medium storing instructions that, when executed by a processor, programs the processor to:
- identify a plurality of words in a string;
- and
- for each word from among the plurality of words:
- for each character in the word for which the number of characters remaining in the word, including the character, is equal to or greater than N, generate a word-bound N-gram based on:
- (1) the character and the next N−1 characters in the word or (2) the character and the remaining characters in the word if the length of the word is less than N such that a last word-bound N-gram for each word includes the last N characters of the word or less than the last N characters of the word;
- and
- generate a feature vector based on the plurality of word-bound N-grams, wherein each dimension of the feature vector comprises a respective word-bound N-gram and a value of each dimension is a corresponding frequency of the respective word-bound N-gram in the string.
Dependencies are inferred from the claim text.