Back to results
US 2019/0102441 A1Published application
Data standardization techniques
AssigneeOracle International Corporation
InventorsMichael Malak, Luis E. Rivas, Mark L. Kreider, Philip Ogren, Robert James Oberbreckling
Life of this patent
Estimating the term…
Abstract
Techniques are disclosed for standardization of data. According to a first technique, standard representation terms are determined for to-be-standardized data using the to-be-standardized data itself and without using any external reference data. According to a second technique, a combination of the to-be-standardized data and an external reference is used to determine standard representation terms for the to-be-standardized data.
Claims20
A non-transitory computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising instructions that when executed by the one or more processors cause the one or more processors to perform processing comprising:
- receiving, by a data standardization system, to-be-standardized data comprising a plurality of terms, wherein each term comprises one or more words;
- determining, by the data standardization system, frequencies of words occurring in the to-be-standardized data;
- identifying, by the data standardization system, a set of one or more common words based upon the frequencies, wherein identifying the set of one or more common words comprises including, in the set of one or more common words, any word that occurs in the to-be-standardized data with a frequency that exceeds a predetermined threshold value;
- determining, by the data standardization system, similarities between the terms in the to-be-standardized data by performing an approximate string matching operation between the terms in the to-be-standardized data, wherein performing the approximate string matching operation between any two terms in the to-be-standardized data comprises matching features extracted from the two terms, and wherein the matching features comprises using a first weight for the matching when a feature corresponds to a common word in the set of one or more common words and using a second weight for the matching when a feature corresponds to a word not in the set of one or more common words, wherein the first weight is different from the second weight;
- determining, by the data standardization system based upon the similarities between the terms in the to-be-standardized data, a plurality of clusters, each cluster in the plurality of clusters comprising one or more terms from the to-be-standardized data;
- for each cluster in the plurality of clusters:
- identifying, by the data standardization system, a recommended term for the cluster from the one or more terms in the cluster;
- and
- identifying, by the data standardization system, a standard representation term for the cluster;
- and
- performing, by the data standardization system, standardization of the to-be-standardized data using standard representation terms identified for the plurality of clusters.
8 more elements
A non-transitory computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising instructions that when executed by the one or more processors cause the one or more processors to perform processing comprising:
- receiving, by a data standardization system, to-be-standardized data comprising a plurality of terms, wherein each term comprises one or more words;
- determining, by the data standardization system, reference data to be used for standardizing the to-be-standardized data;
- determining, by the data standardization system, a set of one or more exact matches and a set of non-exact matches by performing an exact string matching operation between the to-be-standardized data and the reference data;
- determining, by the data standardization system, frequencies of words occurring in the reference data;
- identifying, by the data standardization system, a set of one or more common words based upon the frequencies, wherein identifying the set of one or more common words comprises including, in the set of one or more common words, any word that occurs in the reference data with a frequency that exceeds a predetermined threshold value;
- determining, by the data standardization system, similarities between terms in the set of non-exact matches and terms in the set of one or more exact matches by performing an approximate string matching operation between the terms in the set of non-exact matches and the terms in the set of one or more exact matches, wherein performing the approximate string matching operation between a term in the set of non-exact matches and a term in the set of one or more exact matches comprises matching features extracted from the term in the set of non-exact matches against features extracted from the term in the set of one or more exact matches, and wherein the matching features comprises using a first weight for the matching when a feature corresponds to a common word in the set of one or more common words and using a second weight for the matching when a feature corresponds to a word not in the set of one or more common words, wherein the first weight is different from the second weight;
- determining, by the data standardization system based upon the similarities between the terms in the set of non-exact matches and the terms in the set of one or more exact matches, a suggestion set comprising a plurality of groupings, each grouping in the plurality of groupings comprising terms that share a similarity metric exceeding a first threshold value;
- identifying, by the data standardization system based upon the similarities between the terms in the set of non-exact matches and the terms in the set of one or more exact matches, a subset of the set of non-exact matches comprising one or more terms included in a grouping, of the plurality of groupings, that has a similarity metric not exceeding a second threshold value;
- determining, by the data standardization system, similarities between terms in the subset and terms in the reference data by performing an approximate string matching operation between the terms in the subset and the terms in the reference data, wherein performing the approximate string matching operation between a term in the subset and a term in the reference data comprises matching features extracted from the term in the subset against features extracted from the term in the reference data, and wherein the matching features comprises using the first weight for the matching when a feature corresponds to a common word in the set of one or more common words and using the second weight for the matching when a feature corresponds to a word not in the set of one or more common words;
- determining, by the data standardization system based upon the similarities between the terms in the subset and the terms in the reference data, one or more additional groupings to include in the suggestion set;
- identifying, by the data standardization system, one or more recommended terms from terms in the suggestion set;
- identifying, by the data standardization system, one or more standard representation terms for the suggestion set;
- and
- performing, by the data standardization system, standardization of the to-be-standardized data using the one or more standard representation terms identified for the suggestion set.
11 more elements
A non-transitory computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising instructions that when executed by the one or more processors cause the one or more processors to perform processing comprising:
- receiving, by a data standardization system, to-be-standardized data comprising a plurality of terms, wherein each term comprises one or more words;
- identifying, by the data standardization system, a set of one or more common words based upon the to-be-standardized data;
- determining, by the data standardization system, similarities between the terms in the to-be-standardized data and terms in a set of terms by performing an approximate string matching operation between the terms in the to-be-standardized data and the terms in the set of terms, wherein performing the approximate string matching operation between a term in the to-be-standardized data and a term in the set of terms comprises matching features extracted from the term in the to-be-standardized data against features extracted from the term in the set of terms, and wherein the matching features comprises using a first weight for the matching when a feature corresponds to a word in the set of one or more common words and using a second weight for the matching when a feature corresponds to a word not in the set of one or more common words, wherein the first weight is different from the second weight;
- determining, by the data standardization system based upon the similarities between the terms in the to-be-standardized data and the terms in the set of terms, a plurality of groupings or clusters, each grouping or cluster in the plurality of groupings or clusters comprising one or more terms from the to-be-standardized data;
- identifying, by the data standardization system, one or more recommended terms from terms in the plurality of groupings or clusters;
- identifying, by the data standardization system, one or more standard representation terms for the plurality of groupings or clusters;
- and
- performing, by the data standardization system, standardization of the to-be-standardized data using the one or more standard representation terms identified for the plurality of groupings or clusters.
5 more elements
Dependencies are inferred from the claim text.