Managing data drift and outliers for machine learning models trained for image classification

A device hosting a production environment, comprising:

• a processor;
• and
• a memory coupled with the processor, the memory storing programmed instructions, which when executed by the processor, causes the processor to:
• provide a plurality of digital images to an autoencoder and a Machine Learning (ML) model, wherein the plurality of digital images correspond to one or more types of hardware digital image source devices, wherein the autoencoder and ML model are pre-trained on image training data, wherein the autoencoder reconstructs the plurality of digital images and the ML model classifies the plurality of digital images into one or more categories;
• capture reconstruction errors that occur during reconstruction of the plurality of digital images, wherein the reconstruction errors correspond to losses of information in a reconstructed image compared to a corresponding original image that occur during reconstruction of the plurality of digital images by the autoencoder;
• cluster data points representing the reconstruction errors using an affinity propagation algorithm that allows for a dynamic determination of a quantity of clusters of the data points that result from the clustering;
• determine outliers from the clusters of the data points based on one or more of maximum distance from one or more densely populated clusters, count of values of the data points, and comparison of the values with predefined watermarks in baseline data;
• determine data drift based on changes in densities of the clusters, over a predefined period of time;
• associate classification output of the ML model with the outliers and the data drift, thereby producing a data package;
• provide the data package to a training environment where the data package is used to update the ML model with the classification output associated with the outliers and the data drift, adjust weights of the ML model based on the data package, and update the autoencoder and the baseline data;
• and
• cause an updated ML model that has been updated using the data package to determine a classification category of a new digital image corresponding to the one or more types of digital image source devices.

A device hosting a training environment, comprising:

• one or more processors;
• and
• one or more memories coupled with the one or more processors, the one or more memories storing programmed instructions, which when executed by the one or more processors, causes the one or more processors to:
• train a Machine Learning (ML) model upon a plurality of digital images that correspond to one or more types of hardware digital image source devices;
• receive, from a device hosting a production environment and executing the ML model, a data package that includes classification output of the ML model associated with outliers and data drift,
• wherein:
• the data drift is based on changes in densities of clusters over a pre-determined period of time,
• the clusters are generated via clustering using data points representing reconstruction errors that occur during reconstruction of the plurality of images, the clustering allowing for a dynamic determination of a quantity of clusters of the data points, and
• the reconstruction errors correspond to losses of information in a reconstructed image compared to a corresponding original image that occur during reconstruction of the plurality of images by an autoencoder;
• generate an updated ML model by updating the ML model using the classification output of the ML model associated with the outliers and the data drift, including updating weights of the ML model based on the classification output;
• and
• cause the updated ML model to determine a classification category of a new digital image corresponding to the one or more types of digital image source devices.