Continuously learning, stable and robust online machine learning system

An online machine learning method to be implemented on one or more electronic devices comprising one or more processors, the method comprising:

• receiving, at an Online Preprocessing Engine (OPrE) stored on a non-transitory computer readable memory, streaming data including a set of instances, each instance of the set of instances comprising one or more vectors of inputs that include a plurality of continuous or categorical features produced externally or internally by the OMLS or by actions of one or more of users of the OMLS;
• and (b) discretizing features using the OPrE, wherein the OPrE is capable of imputing missing feature values, normalizing features, detecting drift or change in features, and detecting drift or change in labels for the set of instances, wherein each of the labels is an actual label or a predicted label;
• producing features including engineered features, using an Online Feature Engineering Engine (OFEE) stored on a non-transitory computer readable memory;
• evaluating and selecting features, using an Online Robust Feature Selection Engine (ORFSE) stored on a non-transitory computer readable memory, comprising computing a relevance value of one of the features based on a degree of correlation between a first feature and a second feature using at least one of (i) mutual information-based label relevances between features, (ii) distributional similarities between features, or (iii) similarity of change in distributions of two features to reduce correlations and redundancies between selected features, and performing a statistical significance test to determine a significance of the relevance value that is computed using at least one of (i), (ii), or (iii);
• and
• incorporating and utilizing multiple machine learning algorithms or models utilizing features selected by the ORFSE, at an Online Machine Learning Engine (OMLE) stored on a non-transitory computer readable memory, to generate a result,
• wherein each of the OMLE, the OPrE, the OFEE, and the ORFSE are continuously communicatively coupled to each other;
• wherein the OPrE, which is capable of detecting the drift or change in features, is configured to detect the drift or changes in the plurality of continuous or categorical features, the discretized features, and the engineered features, and
• wherein the method further comprises continuously performing, by the OMLS, operational online machine learning.

An Online Machine Learning System (OMLS) implemented on one or more electronic devices comprising one or more processors, the system comprising:

• an Online Preprocessing Engine (OPrE), stored on a non-transitory computer readable memory and implemented using the one or more processors and configured to (a) receive streaming data including a set of instances, each instance of the set of instances comprising a vector of inputs that includes a plurality of continuous or categorical features produced externally or internally by the OMLS or by actions of one or more of users of the OMLS;
• (b) be able to discretize features;
• (c) be able to impute missing feature values;
• (d) be able to normalize features;
• (e) be able to detect drift or change in features;
• and (f) be able to detect drift or change in labels for the set of instances, wherein each of the labels is an actual label or a predicted label;
• an Online Feature Engineering Engine (OFEE) stored on a non-transitory computer readable memory and implemented using the one or more processors and configured to produce features including engineered features;
• an Online Robust Feature Selection Engine (ORFSE) stored on a non-transitory computer readable memory and implemented using the one or more processors and configured to evaluate and select features by computing a relevance value of one of the features based on a degree of correlation between a first feature and a second feature using at least one of (i) mutual information-based label relevances between features, (ii) distributional similarities between features, or (iii) similarity of change in distributions of two features to reduce correlations and redundancies between selected features, and performing a statistical significance test to determine a significance of the relevance value that is computed using at least one of (i), (ii), or (iii);
• and
• an Online Machine Learning Engine (OMLE) stored on a non-transitory computer readable memory and implemented using the one or more processors and configured to incorporate and utilize one or more machine learning algorithms or models utilizing features selected by the ORFSE to generate a result, and capable of incorporating and utilizing multiple different machine learning algorithms or models;
• an Online Explanation System (OES) stored on a non-transitory computer readable memory and implemented using the one or more processors and configured to provide instance level and model level explanations to a user;
• and
• an Online Human Expert Feedback System (OEFS) stored on a non-transitory computer readable memory and implemented using the one or more processors and configured to obtain, for optimization of operation of the OMLS, instance level expert feedback and model level expert feedback;
• wherein each of the OMLE, the OPrE, the OFEE, the ORFSE, the OES and the OEFS are continuously communicatively coupled to each other and capable of continuous updating;
• wherein the OPrE, which is configured to be able to detect the drift or change in features, is configured to detect the drift or changes in the plurality of continuous or categorical features, the discretized features, and the engineered features, and
• wherein the OMLS is configured to perform continuous online machine learning comprising per-instance learning.

A non-transitory computer readable medium or media containing instructions for executing a method for performing online machine learning using one or more electronic devices comprising one or more processors, the method comprising:

• (a) receiving, at an Online Preprocessing Engine (OPrE), streaming data including a set of instances, each instance of the set of instances comprising a vector of inputs that includes a plurality of continuous or categorical features produced externally or internally by the OMLS or by actions of one or more of users of the OMLS;
• and (b) discretizing features using the OPrE, wherein the OPrE is capable of imputing missing feature values, normalizing features, and detecting drift or change in features and labels for the set of instances, wherein each of the labels is an actual label or a predicted label;
• producing features including engineered features, using an Online Feature Engineering Engine (OFEE);
• evaluating and selecting features, using an Online Robust Feature Selection Engine (ORFSE), comprising computing a relevance value of one of the features based on a degree of correlation between a first feature and a second feature using at least one of (i) mutual information-based label relevances between features, (ii) distributional similarities between features, or (iii) similarity of change in distributions of two features to reduce correlations and redundancies between selected features, and performing a statistical significance test to determine a significance of the relevance value that is computed using at least one of (i), (ii), or (iii);
• and
• incorporating and utilizing multiple machine learning algorithms or models utilizing features, selected by the ORFSE, at an Online Machine Learning Engine (OMLE), to generate a result;
• wherein each of the OMLE, the OPrE, the OFEE and the ORFSE are continuously communicatively coupled to each other;
• wherein the OPrE, which is capable of detecting the drift or change in features, is capable of detecting the drift or changes in the plurality of continuous or categorical features, the discretized features, and the engineered features, and
• wherein the method further comprises continuously operational online machine learning by the OMLS.