System and method for detecting anomalies in prediction generation systems

A computer program product for detecting anomalous activity in online, real-time machine learning environments, the computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to:

• monitor an online scoring environment, wherein the online scoring environment is comprised of machine learning models for scoring incoming requests, the incoming requests being received and thereafter scored by using model learning models, and wherein outcome actions are initiated based on the score received by each incoming request;
• perform statistical analysis on incoming scoring requests to determine whether incoming scoring requests are anomalous, wherein the determination is performed by performing domain checks for categorical variables in the incoming scoring requests, and by detecting out of bound values for continuous variables in the incoming scoring requests;
• determine whether the scores output in the online scoring environment are anomalous, wherein the determination is performed by checking whether null/0 scores are output in the online scoring environment, comparing the output scores against thresholds and historical trends that were established when the model algorithms were trained and/or validated;
• determine whether the outcome actions are anomalous, wherein the determination is performed by determining whether there is a drift between received scores and outcome actions;
• tag the anomalous activity with a flag, the flag comprising information about the type of anomaly that is detected;
• and
• generate an alert if an anomaly is detected in the incoming scoring requests, the scores output, or the outcome actions.