Automatic model monitoring for data streams

A system comprising:

• a processor configured to receive an input dataset;
• use a machine learning model to determine a model score for each data record of at least a portion of the input dataset;
• determine monitoring values, wherein each monitoring value is associated with a measure of similarity between model scores for those data records of the input dataset within a corresponding moving reference window and model scores for those data records of the input dataset within a corresponding moving target window;
• in response to detecting that at least one of the monitoring values meets the threshold, freezing a time location of the reference window and sliding the target window until at least one of the monitoring values falls below the threshold;
• and output the determined monitoring values;
• and a memory coupled to the processor and configured to provide the processor with instructions.

A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

• receiving an input dataset;
• using a machine learning model to determine a model score for each data record of at least a portion of the input dataset;
• determining monitoring values, wherein each monitoring value is associated with a measure of similarity between model scores for those data records of the input dataset within a corresponding moving reference window and model scores for those data records of the input dataset within a corresponding moving target window;
• in response to detecting that at least one of the monitoring values meets the threshold, freezing a time location of the reference window and sliding the target window until at least one of the monitoring values falls below the threshold;
• and
• outputting the determined monitoring values.