System And Method For Fast Identification Of Variable Roles During Initial Data Exploration

A computer-implemented system for determining a role for a data variable for use in data modeling of a physical process, comprising:

• a plurality of data nodes each comprising a node data processor configured to perform operations on a plurality of data records, each data record including a data value for a variable, the plurality of data nodes including non-transitory computer-readable media encoded with a sorted data structure and encoded with data processing instructions, the sorted data structure for storing up to a predetermined number of unique data values for one or more variables, a count for each of the unique data values, and an overflow count for each of the one or more variables,
• the data processing instructions comprising instructions for configuring the data node to:
• determine whether a next data value can be added to the sorted data structure at the data node and that a count associated with that next data value can be added to the sorted data structure when the next data value can be added, determine whether the next data value is already included in the sorted data structure and that the count associated with that next data value can be incremented when the next data value is already included, and determine whether the next data value should not be added to the data structure and that an overflow count at that node should be incremented when the next data value cannot be added;
• wherein one of the data nodes is a root data node comprising a root data processor configured by instructions to:
• communicate data record assignments to the data nodes and a maximum number of levels to record in the sorted data structure, and receive for a plurality of variables a plurality of unique data values, a count for each of the unique data values and an overflow count for the variables;
• wherein a role for a variable can be determined based upon the unique data values, counts and overflow count for a variable.

A computer-program product for performing data mining operations on data, the computer-program product tangibly embodied in a machine-readable non-transitory storage medium and including instructions configured to cause a data processing apparatus to:

• identify to a plurality of node data processors a set of data records containing data values, wherein a particular node data processor is assigned a particular set of data records;
• at the particular node data processor:
• determine whether a data value for a next data record in the particular set of data records can be added to a sorted data structure at the particular node data processor, wherein the particular node data processor is configured for each variable to store up to a predetermined number of unique data values in the sorted data structure and a count for each of the unique data values, and wherein the particular node data processor is configured to store a overflow count of data values that cannot be added to the sorted data structure;
• when the data value can be added and the data value matches a data value in the sorted data structure, increment the count associated with that data value;
• when the data value can be added and the data value does not match a data value in the sorted data structure, add the data value to the sorted data structure;
• when the data value cannot be added, increment the overflow count;
• consolidate the data values and counts for each variable from the particular node data processor with data values and counts from other of the plurality of node data processors into a sorted consolidated data structure and a consolidated overflow count, wherein a role for the variable can be determined based upon the unique data values, counts and overflow count for the variable.

A computer-implemented method of determining a role for a data variable for use in data modeling of a physical process, comprising:

• receiving the identity of a set of data records containing data values and a maximum number of levels to record in a sorted data structure;
• determining for a data variable whether a next data value for a data record can be added to the sorted data structure and that a count associated with that next data value can be added to the sorted data structure when the next data value can be added,
• determining for the data variable whether the next data value is already included in the sorted data structure and that the count associated with that next data value can be incremented when the next data value is already included, and
• determining for the data variable whether the next data value should not be added to the data structure and that an overflow count should be incremented when the next data value cannot be added;
• broadcasting for the data variable a plurality of unique data values, a count for each of the unique data values and an overflow count, wherein the number of unique data values does not exceed the maximum number of levels;
• wherein a role for the variable can be determined based upon the unique data values, counts and overflow count.