System and Method for Data Cleansing

A system for data cleansing, comprising:

• a hardware processor;
• and
• a memory storing instructions executable by the hardware processor for:
• receiving, by a data categorizer one or more data records pre-categorized into one or more categories;
• identifying, by the data categorizer, at least one concept associated with the one or more data records;
• grouping, by the data categorizer, the at least one concept associated with the one or more data records into a plurality of category lists based on the pre-categorized one or more categories associated with each of the one or more data records;
• determining, by the data categorizer, one or more intersection sets based on a comparison between each pair of the plurality of category lists, wherein each intersection set comprises a set of one or more common concepts associated with a pair of category lists;
• and
• replacing, by the data categorizer, each of at least one common concept of the set of one or more common concepts associated with the each intersection set by at least one category name based on an occurrence rate of the each of the at least one common concept of the set of one or more common concepts associated with the each intersection set.

The system of claim wherein replacing the each of the at least one common concept of the set of one or more common concepts associated with the each intersection set further comprises:

• assigning an intensity of confusion value to the one or more intersection sets based on the confusion matrix;
• identifying at least one intersection set from the one or more intersection sets having the intensity of confusion value higher than or equal to an intensity of confusion value of the largest intersection set;
• and
• replacing the each of the at least one common concept of the set of one or more common concepts associated with the at least one intersection set by the at least one category name based on the occurrence rate of the each of the at least one common concept of the set of one or more common concepts associated with the at least one intersection set.

A non-transitory computer-readable medium storing instructions for data cleansing, wherein upon execution of the instructions by one or more processors, the processors perform operations comprising:

• receiving one or more data records pre-categorized into one or more categories;
• identifying at least one concept associated with the one or more data records;
• grouping the at least one concept associated with the one or more data records into a plurality of category lists based on the pre-categorized one or more categories associated with each of the one or more data records;
• determining one or more intersection sets based on a comparison between each pair of the plurality of category lists, wherein each intersection set comprises a set of one or more common concepts associated with a pair of category lists;
• and
• replacing each of at least one common concept of the set of one or more common concepts associated with the each intersection set by at least one category name based on an occurrence rate of the each of the at least one common concept of the set of one or more common concepts associated with the each intersection set.