Method and system for adaptively removing outliers from data used in training of predictive models

A system for generating training data for a machine learning system, comprising:

• a processor;
• and
• a memory for holding programmable code, wherein the programmable code includes instructions which, when executed by the processor, cause the processor to perform a set of acts for collecting data from a clustered database system, the set of acts comprising:
• collecting first data pertaining to a first computing node and second data pertaining to a second computing node in the clustered database system;
• identifying an abnormal state of the first computing node in the clustered database system;
• identifying one or more filter conditions that correspond to the abnormal state, wherein
• the one or more filter conditions are defined with a declaration that includes a signal name, a scope, and an apply period,
• the apply period includes a filter range that comprises an ordered pair, and
• the ordered pair comprises a first component indicative of a first number of datapoints generated before the one or more filter conditions for the abnormal state turn true and a second component indicative of a second number of datapoints generated after the one or more filter conditions for the abnormal state turn false;
• prior to training a predictive model with training data for the clustered database system, performing a set of acts that comprises:
• filtering out a first portion of the first data pertaining to the abnormal state of the first computing node;
• and
• after the second computing node has been determined to be in a normal state during a time period corresponding to the abnormal state of the first computing node, filtering out a second portion of the second data pertaining to the second computing node within the time period during which the second computing node is determined to be in the normal state on a basis that the time period corresponds to the abnormal state of the first computing node;
• and
• training the predictive model at least by providing the predictive model with the training data from which the first portion of the first data and the second portion of the second data have been filtered out.

A computer program product embodied on a non-transitory computer readable medium having stored thereon a sequence of instructions which, when executed by a processor, causes the processor to perform a set of acts, the set of acts comprising:

• collecting first data pertaining to a first computing node and second data pertaining to a second computing node in a clustered database system;
• identifying an abnormal state of the first computing node in the clustered database system;
• identifying one or more filter conditions that correspond to the abnormal state, wherein
• the one or more filter conditions are defined with a declaration that includes a signal name, a scope, and an apply period,
• the apply period includes a filter range that comprises an ordered pair, and
• the ordered pair comprises a first component indicative of a first number of datapoints generated before the one or more filter conditions for the abnormal state turn true and a second component indicative of a second number of datapoints generated after the one or more filter conditions for the abnormal state turn false;
• prior to training a predictive model with training data for the clustered database system, performing a set of acts that comprises:
• filtering out a first portion of the first data pertaining to the abnormal state of the first computing node;
• and
• after the second computing node has been determined to be in a normal state during a time period corresponding to the abnormal state of the first computing node, filtering out a second portion of the second data pertaining to the second computing node within a time period during which the second computing node is determined to be in a normal state on a basis that the time period corresponds to the abnormal state of the first computing node;
• and
• training the predictive model at least by providing the predictive model with the training data from which the first portion of the first data and the second portion of the second data have been filtered out.