Back to results
US 2019/0164080 A1Published application
Auto-scaling hosted machine learning models for production inference
AssigneeAmazon Technologies, Inc.
InventorsStefano STEFANI, Steven Andrew LOEPPKY, Thomas Albert FAULHABER, JR., Craig WILEY, Edo LIBERTY
Life of this patent
Estimating the term…
Abstract
Techniques for auto-scaling hosted machine learning models for production inference are described. A machine learning model can be deployed in a hosted environment such that the infrastructure supporting the machine learning model scales dynamically with demand so that performance is not impacted. The model can be auto-scaled using reactive techniques or predictive techniques.
Claims20
A computer-implemented method comprising:
- receiving a request to automatically scale an amount of model instances within a fleet of one or more model instances in a provider network that host a machine learning model, wherein the request indicates a metric condition that is based on one or more operational metrics associated with the fleet and that indicates when the automatic scaling is to occur;
- monitoring the one or more operational metrics associated with the fleet;
- determining that the metric condition is satisfied;
- and
- adding a model instance to the fleet responsive to the determining that the metric condition is satisfied.
2 more elements
A computer-implemented method comprising:
- monitoring a set of one or more operational metrics associated with a fleet of one or more model instances executing in a provider network that implement a machine learning model;
- determining, based on the monitoring, that a metric condition associated with the fleet is satisfied;
- and
- adding a model instance to the fleet or removing one of the fleet of model instances from the fleet responsive to the determining that the metric condition is satisfied.
1 more element
A system comprising:
- a fleet of one or more model instances executed by a first one or more electronic devices;
- and
- an auto-scaling system implemented by a second one or more electronic devices, wherein the auto-scaling system includes instructions which, when executed by the second one or more electronic devices, cause the auto-scaling system to:
- monitor a set of one or more operational metrics associated with a fleet of one or more model instances executing in a provider network that implement a machine learning model;
- determine, based on the monitoring, that a metric condition associated with the fleet is satisfied;
- and
- add a model instance to the fleet or remove one of the fleet of model instances from the fleet responsive to the determining that the metric condition is satisfied.
4 more elements
Dependencies are inferred from the claim text.