Prognostics and health management service

A computer-implemented method comprising:

• receiving a request to perform model selection;
• evaluating a plurality of models to select model by:
• generating a plurality of metrics for each of the trained plurality of models, the plurality of metrics including at least two of an forewarning time metric of how much in advance of a failure an alert can be raised by the model, event recall metric of how many failure events were alerted to in advance of failure, an event precision metric of a ratio of true and false positives, and an area under a receiver operating characteristic (ROC) curve,
• calculating, for each of the trained plurality of models, a weighted harmonic mean from the at least two metrics, and
• selecting one of plurality of models based on the calculated weighted harmonic means;
• and
• generating and providing a report regarding the selected trained model.

A system comprising:

• a first one or more electronic devices to be managed by a service in a multi-tenant provider network;
• and
• a second one or more electronic devices to implement a model selection service in the multi-tenant provider network, the model selection service including instructions that upon execution cause the model selection service to:
• receive a request to perform model selection,
• evaluate a plurality of models to select model by:
• generate a plurality of metrics for each of the trained plurality of models, the plurality of metrics including at least two of an forewarning time metric of how much in advance of a failure an alert can be raised by the model, event recall metric of how many failure events were alerted to in advance of failure, an event precision metric of a ratio of true and false positives, and an area under a receiver operating characteristic (ROC) curve,
• calculate, for each of the trained plurality of models, a weighted harmonic mean from the at least two metrics, and
• select one of plurality of models based on the calculated weighted harmonic means to use to manage the first one or more electronic devices, and
• generate and provide a report regarding the selected trained model.