Inference service deployment method, device, and storage medium

An electronic device, comprising:

• at least one processor;
• and
• a memory connected in communication with the at least one processor,
• wherein the memory stores an instruction executable by the at least one processor, and the instruction, when executed by the at least one processor, enables the at least one processor to execute:
• obtaining performance information of a runtime environment of a deployment end, wherein the performance information of the runtime environment comprises query time consuming at different quantile levels;
• selecting, according to the performance information of the runtime environment of the deployment end, a target version of an inference service from a plurality of candidate versions of the inference service of a model;
• deploying the target version of the inference service to the deployment end, by:
• determining a resource amount corresponding to the target version of the inference service and a configuration parameter corresponding to the inference service;
• determining a quantity of copies and an external access address of the target version of the inference service;
• and
• launching the target version of the inference service online to the deployment end, based on the resource amount corresponding to the target version of the inference service, the configuration parameter corresponding to the inference service, the quantity of copies and the external access address;
• updating the inference service that has been deployed at the deployment end, by:
• updating a new version of the inference service that has been deployed at the deployment end by a specified quantity of copies in a rolling manner, to gradually replace an old version of the inference service that has been deployed at the deployment end;
• performing data collection for the inference service that has been deployed at the deployment end, to obtain statistical information of a runtime effect of the model, wherein the statistical information comprises at least one of:
• a query data distribution offset of the model in different periods of a model runtime environment, a data distribution offset of the model between query data in the model runtime environment and model training data, or a comparison result of the model between an output result in the model runtime environment and a labeling result;
• and
• judging, according to the statistical information, whether the model is suitable for the model runtime environment.

A non-transitory computer-readable storage medium storing a computer instruction thereon, wherein the computer instruction is configured to cause a computer to execute:

• obtaining performance information of a runtime environment of a deployment end, wherein the performance information of the runtime environment comprises query time consuming at different quantile levels;
• selecting, according to the performance information of the runtime environment of the deployment end, a target version of an inference service from a plurality of candidate versions of the inference service of a model;
• deploying the target version of the inference service to the deployment end, by:
• determining a resource amount corresponding to the target version of the inference service and a configuration parameter corresponding to the inference service;
• determining a quantity of copies and an external access address of the target version of the inference service;
• and
• launching the target version of the inference service online to the deployment end, based on the resource amount corresponding to the target version of the inference service, the configuration parameter corresponding to the inference service, the quantity of copies and the external access address;
• updating the inference service that has been deployed at the deployment end, by:
• updating a new version of the inference service that has been deployed at the deployment end by a specified quantity of copies in a rolling manner, to gradually replace an old version of the inference service that has been deployed at the deployment end;
• performing data collection for the inference service that has been deployed at the deployment end, to obtain statistical information of a runtime effect of the model, wherein the statistical information comprises at least one of:
• a query data distribution offset of the model in different periods of a model runtime environment, a data distribution offset of the model between query data in the model runtime environment and model training data, or a comparison result of the model between an output result in the model runtime environment and a labeling result;
• and
• judging, according to the statistical information, whether the model is suitable for the model runtime environment.