Adaptive filtering and modeling via adaptive experimental designs to identify emerging data patterns from large volume, high dimensional, high velocity streaming data

A system for identifying information in high dimensional data streams having dynamically evolving data patterns, the system comprises:

• one or more processors;
• a memory coupled to the one or more computer processors and comprising instructions, which when performed by the one or more computer processors, cause the one or more processors to perform operations to:
• load at least one a-priori, pre-designed experimental design matrix and at least one modeling operation into memory;
• filter streaming data based on event data continuously to identify diagnostic data points by comparing the event data with the at least one a-priori, pre-designed experimental design matrix;
• and
• perform the modeling operation using the identified diagnostic data points to identify current and emerging patterns of relationships between at least one outcome variable and predictor variables,
• wherein the at least one a-priori, pre-designed experimental design matrix is generated based on combinations of the predictor variables, wherein the combinations are based on an outcome variable, wherein the at least one a-priori, pre-designed experimental design matrix is generated further based on at least one of:
• main effects of predictor variable values;
• limitations of the combinations of predictor variable values;
• constraints of the combination of predictor variable values;
• and
• interaction effects between selected predictor variables.

At least one non-transitory computer readable medium comprising instructions for identifying information in high dimensional streaming data having dynamically evolving data patterns, when executed by at least one processor, cause the at least one processor to perform operations to:

• load at least one a-priori, pre-designed experimental design matrix and at least one modeling operation into memory;
• filter, continuously and real-time, streaming data based on event data to identify diagnostic data points by comparing the event data with the at least one a-priori, pre-designed experimental design matrix;
• and
• perform, continuously and in real-time, the modeling operation using the identified diagnostic data points to identify current and emerging patterns of relationships between at least one outcome variable and predictor variables,
• wherein the at least one a-priori, pre-designed experimental design matrix is generated based on combinations of the predictor variables, wherein the combinations are based on an outcome variable, wherein the at least one a-priori, pre-designed experimental design matrix is generated further based on at least one of:
• main effects of predictor variable values;
• limitations of the combinations of predictor variable values;
• constraints of the combination of predictor variable values;
• and
• interaction effects between selected predictor variables.