Generating a configuration portfolio including a set of model configurations

A system, comprising:

• at least one processor;
• memory in electronic communication with the at least one processor;
• and
• instructions stored in the memory, the instructions being executable by the at least one processor to:
• receive a plurality of model configurations, the plurality of model configurations including a plurality of machine learning models, wherein the plurality of machine learning models are compatible with a plurality of training tasks to predict, for the plurality of training tasks, outputs based on data from the plurality of training tasks;
• generate, during an offline process, a regret matrix for the plurality of model configurations and the plurality of training tasks, the regret matrix including a metric of regret for each model configuration from the plurality of model configurations with respect to accuracy in predicting outputs for each training task from the plurality of training tasks wherein generating the regret matrix comprises, for at least one model configuration of the plurality of model configurations:
• initiating evaluation of a predicted performance of the at least one model configuration with respect to the plurality of training tasks by applying a machine learning model of the at least one model configuration to the plurality of training tasks;
• determining a metric of regret of the at least one model configuration based on the predicted performance of the at least one model configuration with respect to the plurality of training tasks;
• prior to completing evaluation of the predicted performance of the at least one model configuration with respect to the plurality of training tasks, determining that a target metric of regret has been reached;
• and
• stopping evaluation of the predicted performance of the at least one model configuration based on the target metric of regret being reached;
• generate, during the offline process, a configuration portfolio including a subset of model configurations of the plurality of model configurations based on the subset of model configurations having lower metrics of regret than other model configurations from the plurality of model configurations, the subset of model configurations excluding the at least one model configuration;
• and
• provide the configuration portfolio to a computing device for use in selectively identifying an output model configuration from the configuration portfolio during an online process and based on a comparison of metafeatures of an input task and metafeatures of the plurality of training tasks.