Method and apparatus for analyzing coverage, bias, and model explanations in large dimensional modeling data

A system for analyzing coverage, bias and model explanations in large dimensional modeling data, the system comprising:

• at least one programmable processor, and
• a non-transitory, machine-readable medium storing instructions that, when executed by the at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
• discretizing selected variables of a dataset to generate a discretized phase space represented as a grid of a plurality of cells, wherein the dataset comprises a plurality of records;
• applying a grid transformation to a first record in the dataset to assign the first record to a first cell of the plurality of cells of the grid;
• generating a grid index to reference one or more cells using one or more discretized feature vectors;
• creating a grid storage for storing the plurality of records assigned to the one or more cells of the grid;
• creating a coverage storage, separate from a grid storage, for storing data representing a number of records assigned to the one or more cells, wherein coverage storage provides a summary of a count of data points in the plurality of cells in the discretized phase space;
• scoring a record in the dataset using a machine learning model trained on the dataset;
• storing the scores for the one or more cells in a score storage;
• generating score distribution statistics and probability density functions of the scores associated with at least one of a single variable, pair of variables, and triplet of variables;
• assigning explanation tuples for a new scored transaction utilizing distance and probability measures of the score being most probable based on statistics of score probability density functions stored in the grid storage for at least one of the single variable, pair of variables, and triplet of variables;
• providing the explanation tuples, via one or more processors, to help identify biases in the data;
• and
• using the identified biases to automatically adjust a parameter of the machine learning model to mitigate the bias in a subsequent model iteration.