Accelerator based inference service

A computer-implemented method, comprising:

• receiving a request to migrate at least a portion of a machine learning model from a first accelerator portion to a second accelerator portion, the machine learning model to be jointly executed by an accelerator portion and as a part of a client application instance by a non-accelerator portion;
• persisting a state of the first accelerator portion;
• provisioning the second accelerator portion to be attached to the client application instance;
• attaching the second accelerator portion to the client application instance;
• loading at least a portion of the machine learning model, as persisted, onto the attached second accelerator portion;
• after loading the at least a portion of the machine learning model onto the attached second accelerator portion, directing the first accelerator portion to stop taking inference requests;
• detaching the first accelerator portion from the client application instance;
• and
• performing at least a portion of an inference request using the loaded at least a portion of the machine learning model on the second accelerator portion.

A system comprising:

• storage to store at least one machine learning model;
• a plurality of accelerator appliances, each accelerator appliance including at least one accelerator portion;
• a hosted service implemented by a second one or more electronic devices, the hosted service including instructions that upon execution cause the hosted service to:
• receive a request to migrate at least a portion of a machine learning model from a first accelerator portion to a second accelerator portion, the machine learning model to be jointly executed by an accelerator portion and as a part of a client application instance by a non-accelerator portion;
• persist a state of the first accelerator portion;
• provision the second accelerator portion to be attached to the client application instance;
• attach the second accelerator portion to the client application instance;
• load at least a portion of the machine learning model, as persisted, onto the attached second accelerator portion;
• after loading the at least a portion of the machine learning model onto the attached second accelerator portion, direct the first accelerator portion to stop taking inference requests;
• detach the first accelerator portion from the client application instance;
• and
• perform at least a portion of an inference request using the loaded at least a portion of the machine learning model on the second accelerator portion.