Optimizing parameters for machine learning models

A non-transitory computer-readable medium comprising computer program code, the computer program code when executed by a processor of a client device causes the processor to:

• store, by an online system, a plurality of historical datasets, each historical dataset comprising historical parameter values used to train a prior machine learning model, an evaluation score representing a performance of the prior machine learning model, and associated metadata descriptive of the prior machine learning model;
• receive a request to train a machine learning model;
• predict candidate parameter values for training the machine learning model, the candidate parameter values predicted based on a subset of the plurality of historical datasets;
• receive training data for training the machine learning model;
• and
• train the machine learning model using the received training data according to the predicted candidate parameter values.

A method comprising:

• determining an estimated performance score of a trained machine learning model that was trained using candidate parameter values predicted by a prediction model;
• generating a prediction error based on a difference between a predicted occurrence of an event obtained from the trained machine learning model and an actual output;
• determining that a difference between the estimated performance score and the generated prediction error exceeds a threshold error;
• and
• responsive to the determined difference being above the threshold error, triggering a corrective action for the trained prediction model.