Latent feature dimensionality bounds for robust machine learning on high dimensional datasets

A system comprising:

• at least one programmable processor;
• and
• a non-transitory machine-readable medium storing instructions that, when executed by the at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
• monitoring N observed variables, v1 through vN, of a training dataset for a machine learning model, the training dataset having a first dimension;
• translating the N observed variables into m equisized bin indexes which generate mN possible equisized hypercells to estimate a fundamental dimensionality for the training dataset;
• generating one or more samples by assigning a record in the training dataset with numbers j through k as set id;
• generating a merged sample Si, for one or more values of the set id i, where i goes from j to k;
• estimating a fractal dimension of the equisized hypercube phase space based on count of cells with data coverage of at least one data point;
• determining a fundamental dimensionality for the training dataset based on the estimated fractal dimension;
• defining an optimal number of latent features according to the fundamental dimensionality;
• and
• training the machine learning model using the defined optimal number of latent features.

A computer program product comprising a non-transitory machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:

• monitoring, using one or more processors, N observed variables, v1 through vN, of a training dataset for a machine learning model, the training dataset having a first dimension;
• translating the N observed variables into m equisized bin indexes which generate mN possible equisized hypercells to estimate a fundamental dimensionality for the training dataset;
• generating one or more samples by assigning a record in the training dataset with numbers j through k as set id;
• generating a merged sample Si, for one or more values of the set id i, where i goes from j to k;
• and
• estimating a fractal dimension of the equisized hypercube phase space based on count of cells with data coverage of at least one data point;
• determining a fundamental dimensionality for the training dataset based on the estimated fractal dimension;
• adjusting the training dataset's dimensionality from the first dimension to a second dimension;
• and
• training the machine learning model using the training dataset having the second dimension.