Distributed data variable analysis and hierarchical grouping system

A system comprising:

• a controller computing device comprising
• a controller processor;
• and
• a controller computer-readable medium operably coupled to the controller processor, the controller computer-readable medium having controller computer-readable instructions stored thereon that, when executed by the controller processor, cause the controller computing device to
• receive a first policy parameter value that defines a cardinality ratio threshold for identifying the variable as a nominal variable type;
• receive a second policy parameter value that defines a number of unique values threshold for identifying a variable as a high-cardinality variable type;
• receive a third policy parameter value that defines a threshold for a first categorization value;
• request a first computation of a cardinality value and of a number of observations having a non-missing value for each variable of a plurality of variables included in an input dataset by each worker computing device of a plurality of worker computing devices, wherein the input dataset is distributed across the plurality of worker computing devices, wherein the input dataset includes a plurality of observation vectors, wherein each observation vector of the plurality of observation vectors includes a plurality of values, wherein each value of the plurality of values is associated with a different variable to define the plurality of variables, wherein the cardinality value of a variable indicates a number of unique values associated with the variable;
• combine, for each variable of the plurality of variables, the number of observation vectors having the non-missing value computed by each worker computing device of the plurality of worker computing devices;
• combine, for each variable of the plurality of variables, the cardinality value computed by each worker computing device of the plurality of worker computing devices;
• compute a missing rate value for each variable of the plurality of variables using the combined number of observation vectors having the non-missing value and a total number of observation vectors included in the input dataset for each variable of the plurality of variables;
• compute a cardinality ratio value for each variable of the plurality of variables using the combined cardinality value and the combined number of observation vectors having the non-missing value computed for each variable of the plurality of variables;
• for each variable of the plurality of variables,
• compare the computed cardinality ratio value of a respective variable to the received first policy parameter value;
• and
• identify the respective variable as the nominal variable type or as an interval variable type based on the comparison between the computed cardinality ratio value and the received first policy parameter value;
• for each variable of the plurality of variables identified as the nominal variable type,
• compare the combined cardinality value of the respective variable to the received second policy parameter value;
• and
• identify the respective variable as a high-cardinality nominal variable type or as a non-high-cardinality nominal variable type based on the comparison between the combined cardinality value and the received second policy parameter value;
• compare the computed missing rate value of each variable of the plurality of variables to the received third policy parameter value;
• assign each variable of the plurality of variables identified as the high-cardinality nominal variable type to a first category or to a second category based on the comparison between the computed missing rate value and the received third policy parameter value;
• assign each variable of the plurality of variables identified as the non-high-cardinality nominal variable type to a third category or to a fourth category based on the comparison between the computed missing rate value and the received third policy parameter value;
• assign each variable of the plurality of variables identified as the interval variable type to a fifth category or to a sixth category based on the comparison between the computed missing rate value and the received third policy parameter value;
• and
• output an assigned category for each variable of the plurality of variables;
• and
• the plurality of computing devices, each computing device of the plurality of computing devices comprises
• a first processor;
• and
• a first non-transitory computer-readable medium operably coupled to the first processor, the first computer-readable medium having first computer-readable instructions stored thereon that, when executed by the first processor, cause each computing device to
• compute the number of observation vectors having the non-missing value and the cardinality value for each variable of the plurality of variables in response to the first computation request, wherein each respective worker computing device computes the number of observation vectors having the non-missing value and the cardinality value from the subset of the input dataset distributed to the respective worker computing device by reading each observation vector from the subset once.

A method of providing analysis of data and grouping of variables in support of analytics, the method comprising:

• receiving a first policy parameter value that defines a cardinality ratio threshold for identifying the variable as a nominal variable type;
• receiving a second policy parameter value that defines a number of unique values threshold for identifying a variable as a high-cardinality variable type;
• receiving a third policy parameter value that defines a threshold for a first categorization value;
• requesting, by a computing device, a first computation of a cardinality value and of a number of observations having a non-missing value for each variable of a plurality of variables included in an input dataset by each worker computing device of a plurality of worker computing devices, wherein the input dataset is distributed across the plurality of worker computing devices, wherein the input dataset includes a plurality of observation vectors, wherein each observation vector of the plurality of observation vectors includes a plurality of values, wherein each value of the plurality of values is associated with a different variable to define the plurality of variables, wherein the cardinality value of a variable indicates a number of unique values associated with the variable;
• computing, by each worker computing device of the plurality of worker computing devices, the number of observation vectors having the non-missing value and the cardinality value for each variable of the plurality of variables in response to the first computation request, wherein each respective worker computing device computes the number of observation vectors having the non-missing value and the cardinality value from a subset of the input dataset distributed to the respective worker computing device by reading each observation vector from the subset once;
• combining, by the computing device, for each variable of the plurality of variables, the number of observation vectors having the non-missing value computed by each worker computing device of the plurality of worker computing devices;
• combining, by the computing device, for each variable of the plurality of variables, the cardinality value computed by each worker computing device of the plurality of worker computing devices;
• computing, by the computing device, a missing rate value for each variable of the plurality of variables using the combined number of observation vectors having the non-missing value and a total number of observation vectors included in the input dataset for each variable of the plurality of variables;
• computing, by the computing device, a cardinality ratio value for each variable of the plurality of variables using the combined cardinality value and the combined number of observation vectors having the non-missing value computed for each variable of the plurality of variables;
• for each variable of the plurality of variables,
• comparing, by the computing device, the computed cardinality ratio value of a respective variable to the received first policy parameter value;
• and
• identifying, by the computing device, the respective variable as the nominal variable type or as an interval variable type based on the comparison between the computed cardinality ratio value and the received first policy parameter value;
• for each variable of the plurality of variables identified as the nominal variable type,
• comparing, by the computing device, the combined cardinality value of the respective variable to the received second policy parameter value;
• and
• identifying, by the computing device, the the respective variable as a high-cardinality nominal variable type or as a non-high-cardinality nominal variable type based on the comparison between the combined cardinality value and the received second policy parameter value;
• comparing, by the computing device, the computed missing rate value of each variable of the plurality of variables to the received third policy parameter value;
• assigning, by the computing device, each variable of the plurality of variables identified as the high-cardinality nominal variable type to a first category or to a second category based on the comparison between the computed missing rate value and the received third policy parameter value;
• assigning, by the computing device, each variable of the plurality of variables identified as the non-high-cardinality nominal variable type to a third category or to a fourth category based on the comparison between the computed missing rate value and the received third policy parameter value;
• assigning, by the computing device, each variable of the plurality of variables identified as the interval variable type to a fifth category or to a sixth category based on the comparison between the computed missing rate value and the received third policy parameter value;
• and
• outputting, by the computing device, an assigned category for each variable of the plurality of variables.