Back to results
US 11,899,650 B2Granted
Using statistical dispersion in data process generation
AssigneeCapital One Services, LLC
InventorsGunther Havel, Ashwin Assysh Sharma
Life of this patent
Estimating the term…
Abstract
Methods and systems are described herein for facilitating data integrity processes using measures of statistical dispersion (e.g., gini impurities) of dataset features. The described mechanism may be also be used for selection and dimensionality reduction. Dimensionality reduction may enable storing the dataset using less storage space or performing other operations on the dataset using less resources. In some embodiments, the above described mechanism may be used for supervised categorial clustering and/or categorical classification.
Claims17
A system for facilitating data integrity processes, the system comprising:
- one or more processors;
- and
- a non-transitory computer-readable storage medium storing instructions, which when executed by the one or more processors cause the one or more processors to:
- access a database table comprising plurality of columns and a plurality of rows;
- select a target column of the plurality of columns and a plurality of candidate columns of the plurality of columns;
- determine, for each column of the plurality of candidate columns, a first corresponding gini impurity value based on field values in the target column;
- select a first column of the plurality of candidate columns based on the first column having a lowest first gini impurity value of the first corresponding gini impurity values;
- in response to determining that the lowest first gini impurity value is not zero:
- determine, for each column of the plurality of candidate columns other than the first column, a second corresponding gini impurity value based on the field values in the target column;
- and
- select a second column of the plurality of candidate columns based on the first column having a lowest second gini impurity value of the second corresponding gini impurity values;
- and
- in response to determining that the lowest second gini impurity value is zero, generate a data integrity rule for a target feature comprising a mapping that indicates an association between an identifier of the target column and a combination of first and second identifiers of the first column and the second column, wherein the data integrity rule indicates that a row includes valid data when a value in the target column is defined by the combination of values in the first column and the second column.
10 more elements
A method comprising:
- selecting, from a plurality of features in a dataset, a target feature, and a plurality of candidate features;
- determining, for each of the plurality of candidate features, a corresponding first measure of statistical dispersion, based on the target feature for a corresponding set of records;
- selecting a first feature having a lower first measure of statistical dispersion than other features in the plurality of candidate features;
- in response to determining that the lower first measure of statistical dispersion is not zero:
- determining, for each combination of the first feature and another feature of the plurality of candidate features, a corresponding second measure of statistical dispersion based on the target feature for the corresponding set of records;
- and
- selecting a second feature having a lower second measure of statistical dispersion in combination with the first feature than any other feature combination;
- and
- in response to determining that the lower second measure of statistical dispersion satisfies a threshold, generating a data integrity rule for the target feature comprising a mapping that indicates an association between an identifier of the target feature and a combination of first identifier associated with the first feature and a second identifier associated with the second feature, wherein the data integrity rule indicates that a record includes valid data when a value in the target feature is defined by the combination of values of the first feature and the second feature.
6 more elements
A non-transitory, computer-readable medium, storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
- selecting, from a plurality of features in a dataset, a target feature, and a plurality of candidate features;
- determining, for each of the plurality of candidate features, a corresponding first measure of statistical dispersion, based on the target feature for a corresponding set of records;
- selecting a first feature having a lower first measure of statistical dispersion than other features in the plurality of candidate features;
- in response to determining that the lower first measure of statistical dispersion is not zero:
- determining, for each combination of the first feature and another feature of the plurality of candidate features, a corresponding second measure of statistical dispersion based on the target feature for the corresponding set of records;
- and
- selecting a second feature having a lower second measure of statistical dispersion in combination with the first feature than any other feature combination;
- and
- in response to determining that the lower second measure of statistical dispersion satisfies a threshold, generating a data integrity rule for the target feature comprising a mapping that indicates an association between an identifier of the target feature and a combination of a first identifier associated with the first feature and a second identifier associated with the second feature, wherein the data integrity rule indicates that a record includes valid data when a value in the target feature is defined by the combination of values of the first feature and the second feature.
6 more elements
Dependencies are inferred from the claim text.