Perturbed records generation
AssigneeInternational Business Machines Corporation
InventorsLukasz G. Cmielowski, Rafal Bigaj, Maksymilian Erazmus, Wojciech Sobala
Life of this patent
Estimating the term…
Abstract
Reducing a count of perturbed records in a machine learning dataset by application of a correlation matrix of feature values identified in training records to reduce the number of features represented in the perturbed records. Deleting one of a pair of correlated records is achieved with reference to a correlation score that identifies features of sufficient similarity to be paired up. Reducing the number of features for which values are assigned in a data perturbation process results in a relatively reduced number of perturbed records.
Claims20
A computer-implemented method for feature correlation supporting efficient data perturbation for testing machine learning models, the method comprising:
- obtaining a set of structured data including individual records having a set of data features and corresponding data values, wherein processing the set of structured data by a machine learning model results in a proposed action corresponding to an individual record;
- generating a correlation matrix incorporating the set of data features and the corresponding data values;
- identifying, with reference to the correlation matric, a set of data feature pairs as duplicative of one another according to a correlation criterion;
- removing, from the set of data features, one data feature of each identified data feature pair to generate a reduced dataset;
- generating a set of perturbed records for arbitrary indicator identification based on value variances for a target data feature of the reduced dataset, the arbitrary indicator identification being a test of the machine learning model;
- and.
- determining, according to the set of perturbed records, a set of arbitrary indicators associated with the proposed action corresponding to the individual record.
A computer program product for feature correlation in data perturbation, the computer program product comprising:
- a set of storage device(s);
- and
- computer code stored collectively in the set of storage device(s), with the computer code including data and instructions to cause a processor(s) set to perform at least the following operations:
- obtaining a set of structured data including individual records having a set of data features and corresponding data values, wherein processing the set of structured data by a machine learning model results in a proposed action corresponding to an individual record;
- generating a correlation matrix incorporating the set of data features and the corresponding data values;
- identifying, with reference to the correlation matric, a set of data feature pairs as duplicative of one another according to a correlation criterion;
- removing, from the set of data features, one data feature of each identified data feature pair to generate a reduced dataset;
- generating a set of perturbed records for arbitrary indicator identification based on value variances for a target data feature of the reduced dataset, the arbitrary indicator identification being a test of the machine learning model;
- and
- determining, according to the set of perturbed records, a set of arbitrary indicators associated with the proposed action corresponding to the individual record.
A computer system for feature correlation in data perturbation, the computer system comprising:
- a processor(s) set;
- a set of storage device(s);
- and
- computer code stored collectively in the set of storage device(s), with the computer code including data and instructions to cause the processor(s) set to perform at least the following operations:
- obtaining a set of structured data including individual records having a set of data features and corresponding data values, wherein processing the set of structured data by a machine learning model results in a proposed action corresponding to an individual record;
- generating a correlation matrix incorporating the set of data features and the corresponding data values;
- identifying, with reference to the correlation matric, a set of data feature pairs as duplicative of one another according to a correlation criterion;
- removing, from the set of data features, one data feature of each identified data feature pair to generate a reduced dataset;
- generating a set of perturbed records for arbitrary indicator identification based on value variances for a target data feature of the reduced dataset, the arbitrary indicator identification being a test of the machine learning model;
- and.
- determining, according to the set of perturbed records, a set of arbitrary indicators associated with the proposed action corresponding to the individual record.
Dependencies are inferred from the claim text.