Optimizing machine learning based on embedding smart data drift

A system for optimizing a machine learning model, the system comprising:

• a processor and an associated memory, the processor being configured for:
• obtaining one or more embedding vectors based on a prediction of the machine learning model;
• mapping the embedding vectors from a higher dimensional space to a two- or three-dimensional space to generate one or more high density points in the two- or three-dimensional space, each high-density point including at least one of the embedding vectors;
• clustering the high-density points by running a clustering algorithm multiple times, each time with a different set of parameters to generate one or more clusters;
• applying a purity metric to each cluster to generate a normalized purity score of each cluster;
• identifying one or more clusters with a normalized purity score lower than a threshold;
• and
• optimizing the machine learning model based on the identified one or more clusters.