System and method for processing a large data set using a prediction model having a feature selection capability
AssigneeMicrosoft Corporation
InventorDavid Earl Heckerman
Life of this patent
Estimating the term…
Abstract
The present invention includes a system and a method for processing large data sets that are too large to conveniently fit into a formal database application. The large data set processing system and method use a prediction model having a feature selection capability to process a fraction of the large data set and define useful predictors. The useful features are used to make predictions for the entire data set. The large data set processing system includes a useful predictor module, for defining useful predictors, and a feature-selection prediction model, for processing a portion of the data set (including the useful predictors) to obtain prediction results.
Claims35
A method for processing a data set containing samples and features, comprising:
- loading a portion of the samples and each of the features into a database application;
- defining useful predictors using a feature-selection prediction model, the useful predictors corresponding to a portion of the features;
- loading the useful predictors and each of the samples into a database application;
- and
- determining prediction results by processing the feature-selection prediction model against the samples and useful predictors contained in the database application;
- wherein the data set is too large for all of the samples and all of the features to be loaded into the database application simultaneously.
A method for processing a large set of data containing samples and features, the large data set being unable to fit into a database application, comprising:
- building a prediction model having a feature selection capability using each of the features and a randomly-selected fraction of the samples;
- using the feature-selection prediction model to define a portion of the features as useful predictors based on how helpful each feature is in making a desired prediction;
- and
- processing each of the samples and the useful predictors using the feature-selection prediction model to determine prediction results for the desired prediction.
A computer-readable medium having computer-executable instructions for processing a data set containing samples and features that cannot be loaded into a database application, comprising:
- obtaining a first subset containing a fraction of the data set including each of the features;
- building a feature-selection prediction model using the first subset;
- defining useful predictors using the feature-selection prediction model, the useful predictors corresponding to features that are helpful and useful in making a desired prediction;
- obtaining a second subset containing a fraction of the data set including those features defined as useful predictors;
- processing the second subset using the feature-selection prediction model to obtain prediction results of the desired prediction for each of the samples.
A method for processing a large data set containing consumers and features to predict which consumers will be interested in a target product, the large data set containing more data than can be loaded into a database application, comprising:
- selecting each of the features and randomly selecting a number of consumers corresponding a fraction of the consumers to obtain a first subset of the large data set;
- building a feature-selection prediction model using the first subset;
- using the feature-selection prediction model to select features that are useful at predicting which consumers will be interested in the target product and defining those features as useful predictors;
- selecting the useful predictors and each of the consumers to obtain a second subset of the large data set;
- and
- processing the second subset using the feature-selection prediction model to predict which consumers will be interested in the target product.
A large data set processing system for processing a large data set containing samples and features, the large data set containing an amount of data that cannot fit into a database application, comprising:
- a prediction model having a feature selection capability that allows selection and control of the number of features used for processing;
- a useful predictor module that defines useful predictors, the useful predictors being a portion of the features and being useful and relevant to processing;
- and
- prediction results obtained by processing the useful predictors and each of the samples.
A large data set processing system for predicting which consumers will purchase a target product, comprising:
- a large data set containing consumers and features, the large data set containing more data than can be loaded into a database application;
- a first subset containing a randomly-selected portion of the consumers and each of the features;
- a prediction model having a feature selection capability for defining as useful predictors a portion of the features that are useful and relevant in making the prediction;
- a second subset containing the useful predictors and each of the consumers;
- and
- a target group containing those consumers that are predicted to purchase the target product as determined by having the feature-selection prediction model process the second subset.
Dependencies are inferred from the claim text.