US 2025/0371384 A1Published application

Swapping models based on inference request monitoring

AssigneeSnowflake Inc.

InventorsVincent Chan, Xu Chen, Pawel Pollak, Zhaotian Wang, Hao Zhang

Life of this patent

Estimating the term…

Loading patent…

Abstract

Various embodiments described herein provide for systems, methods, devices, instructions, and like for swapping artificial intelligence models, such as large language models (LLMs), based on inference request monitoring. In particular, some embodiments monitor inference requests submitted to various inference engines (where each inference engine comprises a group of software containers sharing assigned computing resources) and, based on analysis of inference request data, available models, currently loaded models, or a combination thereof, determine whether to swap out a set of AI models currently active on a select inference engine with another set of AI models available on the select inference engine.

Claims30

Dependent claims

A system comprising:

at least one hardware processor;
and
at least one memory storing instructions that cause the at least one hardware processor to perform operations comprising:
monitoring inference requests submitted to one or more inference engines, each inference engine having an assigned set of computing resources, the one or more inference engines comprising a select inference engine;
for each individual inference engine of the one or more inference engines:
monitoring one or more available artificial intelligence models that are available for loading on the individual inference engine;
and
monitoring one or more loaded artificial intelligence models currently loaded on the individual inference engine to service inference requests sent to the individual inference engine;
based on the monitoring of the inference requests, the monitoring of the one or more available artificial intelligence models, and the monitoring of the one or more loaded artificial intelligence models, determining whether to swap a first set of artificial intelligence models currently loaded on the select inference engine with a second set of artificial intelligence models from the one or more available artificial intelligence models, the second set of artificial intelligence models not being currently loaded on the select inference engine;
and
in response to determining to swap the first set of artificial intelligence models on the select inference engine with the second set of artificial intelligence models, causing the select inference engine to load the second set of artificial intelligence models on the select inference engine in place of the first set of artificial intelligence models.

8 more elements

A method comprising:

monitoring, by a hardware processor, inference requests submitted to one or more inference engines, each inference engine having an assigned set of computing resources, the one or more inference engines comprising a select inference engine;
for each individual inference engine of the one or more inference engines:
monitoring one or more available artificial intelligence models that are available for loading on the individual inference engine;
and
monitoring one or more loaded artificial intelligence models currently loaded on the individual inference engine to service inference requests sent to the individual inference engine;
based on the monitoring of the inference requests, the monitoring of the one or more available artificial intelligence models, and the monitoring of the one or more loaded artificial intelligence models, determining, by the hardware processor, whether to swap a first set of artificial intelligence models currently loaded on the select inference engine with a second set of artificial intelligence models from the one or more available artificial intelligence models, the second set of artificial intelligence models not being currently loaded on the select inference engine;
and
in response to determining to swap the first set of artificial intelligence models on the select inference engine with the second set of artificial intelligence models, causing, by the hardware processor, the select inference engine to load the second set of artificial intelligence models on the select inference engine in place of the first set of artificial intelligence models.

5 more elements

Dependencies are inferred from the claim text.