Fault-tolerant accelerator based inference service

A computer-implemented method, comprising:

• attaching a first set of one or more accelerator slots of an accelerator appliance to an application instance according to an application instance configuration, the attached application instance attached to the accelerator appliance in a multi-tenant provider network, the accelerator appliance comprising a plurality of accelerators, the plurality of accelerators having a compute capacity, each accelerator slot of the first set of one or more accelerator slots corresponding to a fraction of the compute capacity of the plurality of accelerators;
• loading a machine learning model of the attached application instance onto the first set of one or more accelerator slots;
• handling a first set of one or more inference calls, made by an application of the attached application instance to the loaded machine learning model, using the first set of one or more accelerator slots;
• and
• based on determining that the first set of one or more accelerator slots no longer meet at least one of a timing requirement or a cost requirement, migrating processing for the attached application instance from the first set of one or more accelerator slots to a second set of one or more accelerator slots including detaching the first set of one or more accelerator slots from the application instance, attaching the second set of one or more accelerator slots to the application instance, and loading the machine learning model of the attached application instance onto the second set of one or more accelerator slots;
• handling a second set of inference calls, made by the application of the attached application instance to the loaded machine learning model, using the second set of one or more accelerator slots;
• and
• returning, to the application of the application instance, a result of the second set of one or more inference calls.

A system comprising:

• storage to store an application, the application including a machine learning model;
• and
• one or more electronic devices to implement an elastic inference service, the elastic inference service including an application instance and an accelerator appliance, the elastic inference service to:
• attach a first set of one or more accelerator slots of the accelerator appliance to the application instance of according to an application instance configuration, the attached application instance remotely attached to the accelerator appliance in a multi-tenant provider network, the accelerator appliance comprising a plurality of accelerators, the plurality of accelerators having a compute capacity, each accelerator slot of the first set of one or more accelerator slots corresponding to a fraction of the compute capacity of the plurality of accelerators;
• load the machine learning model onto the first set of one or more accelerator slots;
• handle a first set of inference calls, made by an application of the attached application instance, using the first set of one or more accelerator slots;
• determine that the first set of one or more accelerator slots no longer meet at least one of a timing requirement or a cost requirement;
• migrate processing for the attached application instance from the first set of one or more accelerator slots to a second set of one or more accelerator slots including detaching the first set of one or more accelerator slots from the application instance, attaching the second set of one or more accelerator slots to the application instance, and loading the machine learning model of the attached application instance onto the second set of one or more accelerator slots;
• handle a second set of one or more inference calls, made by the application of the attached application instance to the loaded machine learning model, using the second set of one or more accelerator slots;
• and
• return, to the application of the application instance, a result of the second set of one or more inference calls.