US 2020/0089650 A1Published application

Techniques for automated data cleansing for machine learning algorithms

AssigneeSoftware AG

InventorsSwapnil SHARMA, Thanikachalam SUBRAMANIAN, SrinivasaRaju GOTTIMUKKALA

Life of this patent

Estimating the term…

Loading patent…

Abstract

Machine learning models typically are based on processing large-volume datasets, and datasets are preprocessed so that the machine learning can provide sound results. In building a model, certain example embodiments generate meta-features for each of a number of independent variables in an accessed portion of the dataset. The meta-features are provided as input to pre-trained classification models. Those models output, for the independent variables, indications of one or more appropriate missing value imputation operations, and one or more appropriate other preprocessing data cleansing related operations. The data in the dataset is transformed by selectively applying the missing value imputation operation(s) and the other preprocessing operation(s), in accordance with the independent variables associated with the data, thereby performing the preprocessing in an automated and programmatic way that helps improve the quality of the built model. Ultimately, queries received over a computer-mediated interface can be processed using the built machine learning model.

Claims22

Dependent claims

A machine learning system, comprising:

a non-transitory computer readable storage medium storing thereon a dataset having data from which a machine learning model is buildable;
an electronic computer-mediated interface configured to receive a query processable in connection with a machine learning model;
processing resources including at least one hardware processor operably coupled to a memory, the processing resources being configured to execute instructions to perform functionality comprising:
accessing at least a portion of the dataset;
for each of a plurality of independent variables in the accessed portion of the dataset:
generating meta-features for the respective independent variable;
providing, as input to at least first and second pre-trained classification models that are different from one another, the generated meta-features for the respective independent variable;
receiving, as output from the first pre-trained classification model, an indication of one or more missing value imputation operations appropriate for the respective independent variable;
and
receiving, as output from the second pre-trained classification model, an indication of one or more other preprocessing data cleansing related operations appropriate for the respective independent variable;
transforming the data in the dataset by selectively applying to the data the one or more missing value imputation operations and the one or more other preprocessing data cleansing-related operations, in accordance with the independent variables associated with the data;
building the machine learning model based on the transformed data;
and
enabling queries received over the electronic interface to be processed using the built machine learning model.

11 more elements

A method of configuring a machine learning system, the method comprising:

accessing at least a portion of a dataset having data from which a machine learning model is buildable;
for each of a plurality of independent variables in the accessed portion of the dataset, and using at least one processor:
generating meta-features for the respective independent variable;
providing, as input to at least first and second pre-trained classification models that are different from one another, the generated meta-features for the respective independent variable;
receiving, as output from the first pre-trained classification model, an indication of one or more missing value imputation operations appropriate for the respective independent variable;
and
receiving, as output from the second pre-trained classification model, an indication of one or more other preprocessing data cleansing related operations appropriate for the respective independent variable;
transforming the data in the dataset by selectively applying to the data the one or more missing value imputation operations and the one or more other preprocessing data cleansing-related operations, in accordance with the independent variables associated with the data;
building the machine learning model based on the transformed data;
and
enabling queries received over a computer-mediated interface to be processed using the built machine learning model.

8 more elements

A non-transitory computer-readable storage medium tangibly storing a program, when executed by a processor of a computing system, performs instructions comprising:

accessing at least a portion of a dataset having data from which a machine learning model is buildable;
for each of a plurality of independent variables in the accessed portion of the dataset, and using at least one processor:
generating meta-features for the respective independent variable;
providing, as input to at least first and second pre-trained classification models that are different from one another, the generated meta-features for the respective independent variable;
receiving, as output from the first pre-trained classification model, an indication of one or more missing value imputation operations appropriate for the respective independent variable;
and
receiving, as output from the second pre-trained classification model, an indication of one or more other preprocessing data cleansing related operations appropriate for the respective independent variable;
transforming the data in the dataset by selectively applying to the data the one or more missing value imputation operations and the one or more other preprocessing data cleansing-related operations, in accordance with the independent variables associated with the data;
building the machine learning model based on the transformed data;
and
enabling queries received over a computer-mediated interface to be processed using the built machine learning model.

8 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/131,125Filed 14 Sep 2018
Published: 19 Mar 2020
Term ends: Not estimatedThe source records don’t support an estimate.

Classification8 codes

G06F 16/215
Design, administration or maintenance of databases: improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or…
G06F 16/245
Querying: query processing
G06F 18/15
Pre-processing; Data cleansing: statistical pre-processing, e.g. techniques for normalisation or restoring missing data
G06F 18/214
Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation: generating training…