US 10,372,572 B1Granted

Prediction model testing framework

AssigneeAmazon Technologies, Inc.

InventorsNitin Kesarwani, Sharon Xuelun Yang, Wei Huang

Life of this patent

Estimating the term…

Loading patent…

Abstract

A prediction model testing system includes a test environment that is used to test a prediction model under test (PMUT). A metrics collector in a production environment collects and stores production metrics data generated from computing resources in a production environment. A production predictor in the production environment generates production predictions for the metrics, using a production prediction model. A test manager may make the production metrics data available in a test environment. Test predictions are generated in the test environment from the metrics data using the PMUT. The test manager may then calculate respective prediction errors of the production prediction model and the PMUT, and generate a report indicating the differences between the two sets of prediction errors. The report may be used by the test management system to determine whether a test of the PMUT was successful.

Claims20

Dependent claims

A system comprising:

a metrics collector implemented by one or more computing devices comprising one or more hardware processors and memory configured to:
capture data for one or more metrics generated from one or more computing resources in a production environment of a service provider network;
and
store the captured data in the production environment;
a production predictor implemented by one or more computing devices comprising one or more hardware processors and memory configured to:
generate, in the production environment, production predictions for the one or more metrics based at least in part on a production prediction model and the captured data;
and
a test manager implemented by one or more computing devices comprising one or more hardware processors and memory configured to:
provide the captured data in a test environment;
generate, in the test environment, test predictions for at least a subset of the one or more metrics based at least in part on the prediction model under test (PMUT) and the captured data;
and
obtain observed data for the one or more metrics from the production environment corresponding to the test predictions;
determine test prediction errors between the test predictions and observed data for the one or more metrics from the production environment corresponding to the test predictions;
determine production prediction errors between the production predictions and the observed data;
and
generate a report indicating differences between the test prediction errors and the production prediction errors.

13 more elements

A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors cause the one or more processors to:

providing captured data from a production data store, the captured data comprising data for one or more metrics generated from operation of one or more computing resources in a production environment;
generate, in a test environment, one or more test predictions for at least a subset of the one or more metrics based at least in part on a prediction model under test (PMUT) and the captured data;
and
obtaining observed data for the one or more metrics from the production environment corresponding to the test predictions;
determining test prediction errors between the test predictions and the observed data;
determine production prediction errors between production predictions and the observed data, the production predictions generated in the production environment based at least in part on a production prediction model and the captured data;
and
generating a report indicating differences between the test predictions and the production predictions.

5 more elements

Dependencies are inferred from the claim text.