Data Analysis Computer System and Method For Parallelized and Modularized Analysis of Big Data

A computer-implemented system and method for parallel or distributed predictive, causal and feature selection analysis of Big Data consisting of the following steps:

• a. initializing parameter max-k (maximum size of the conditioning set) with a non-negative integer, as defined by a user;
• b. initializing M with an empty set;
• c. initializing E with all variables excluding T;
• d. initializing the conditioning subset C[1] with an empty set;
• e. repeating the following steps e.i-e.vii until the exit condition is met;
• i. iterating in parallel or distributed fashion over variables X in E and conditioning subsets in C;
• 1. if T becomes statistically independent of X given some conditioning subset C[i], removing X from E;
• ii. if max-k is equal to zero, assigning E to M, outputting M, and terminating;
• iii. if E is empty, exiting from the iterative loop and proceeding to step f below;
• iv. selecting a variable Y that maximizes association with T over variables in E;
• v. assigning to C all subsets of M of size up to max-k-1, union with the variable Y;
• vi. adding Y to M;
• vii. removing Y from E;
• f. iterating in parallel or distributed fashion over variables X in M;
• i. assigning to C all non-empty subsets of M excluding X of size up to max-k;
• 1. iterating in parallel or distributed fashion over conditioning subsets in C;
• a) if T becomes statistically independent of X given some conditioning subset C[i], removing X from M;
• and
• g. outputting M.

A computer-implemented system and method for parallel or distributed predictive, causal and feature selection analysis of Big Data consisting of the following steps:

• a. initializing parameter max-k (maximum size of the conditioning set) with a non-negative integer, as defined by a user;
• b. initializing M with an empty set;
• c. initializing E with all variables excluding T;
• d. initializing the conditioning subset C[1] with an empty set;
• e. repeating the following steps e.i-e.ix until the exit condition is met;
• i. initializing R with an empty set;
• ii. iterating in parallel or distributed fashion over conditioning subsets in C;
• 1. Adding to R all variables X from E that are rendered statistically independent of T given a conditioning subset C[i];
• iii. removing from E variables R;
• iv. if max-k is equal to zero, assigning E to M, outputting M, and terminating;
• v. if E is empty, exiting from the iterative loop and proceeding to step f below;
• vi. selecting a variable Y that maximizes association with T over variables in E;
• vii. assigning to C all subsets of M of size up to max-k-1, union with the variable Y;
• viii. adding Y to M;
• ix. removing Y from E;
• f. assigning to C all non-empty subsets of M excluding X of size up to max-k;
• g. assigning an empty set to R;
• h. iterating in parallel or distributed fashion over all conditioning subsets in C;
• i. Adding to R all variables X from M excluding the conditioning subset C[i] that are rendered statistically independent of T given the conditioning subset C[i];
• i. removing R from M;
• and
• j. outputting M.