# US 2026/0205422 A1 · Dynamic load balancer and health monitor for artificial intelligence model deployments

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0205422 A1
- Document ID: US20260205422A1
- Status: Published application
- Kind: A1
- Application number: 19/017,280
- Filed: 2025-01-10
- Published: 2026-07-16
- Earliest claimed: 2025-01-10
- Assignees: Bold Limited
- Applicants: BOLD Limited
- Inventors: Umar Afzal HAFIZ; Vandana SINGH; Baroon ANAND
- Web page: https://patentgrep.com/patent/US20260205422A1

## CPC

- H04L 47/623: {Weighted service order}
- H04L 41/16: using machine learning or artificial intelligence
- H04L 47/125: by balancing the load, e.g. traffic engineering
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Certain aspects of the disclosure provide a computer-implemented method for monitoring the health of artificial intelligence (AI) model deployments and provide load balancing of user requests to the AI models. The method periodically sends health-status checks to identify available AI models and unavailable AI models. The method sends request obtained from users to the available AI models that corresponds to an AI model types identified in the requests and within rate limits associated with the available AI models. The method sends answers generated by the available AI models to the users.

## Claims

### Claim 1

A computer-implemented method, comprising:
periodically sending health-status checks to a vendor application programming interface (API);
obtaining as output from the vendor API a health-status response identifying an available AI model;
sending a request obtained from a client to the available AI model that corresponds to an AI model type identified in the request and is within a rate limit associated with the available AI model; and
sending an answer generated by the available AI model to the client.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
fetching current instances of the AI models in a group of AI models from an AI models database; and
identifying available AI models in the group of AI models, type of AI models in the group of AI models, geographical region of the group of AI models, and vendor of the group of AI models.

### Claim 3 (depends on 1)

The method of claim 1, wherein periodically sending the health-status check to the vendor API comprises sending a health-status check to the vendor API at regular intervals.

### Claim 4 (depends on 1)

The method of claim 1, wherein obtaining as output from the vendor API the health-status response identifying the available AI model comprises:
receiving the health-status response from the vendor API, the health-status response identifies each AI model in a group of AI models as an available AI model if the AI model is active or an unavailable AI model if the AI model not active;
updating an AI model health status data store with information identifying available AI models and unavailable AI models based on the health-status response; and
automatically sending an event-status report to stakeholders, the event-status report identifying the available AI models and the unavailable AI models in the group of AI models.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
for each respective unavailable AI model of the group of AI models:
sending health-status checks to the vendor API at shortened intervals until the vendor API sends a health-status response that identifies the respective unavailable AI model as an available AI model,
wherein the shortened intervals have a shorter duration than regular intervals between health-status checks.

### Claim 6 (depends on 1)

The method of claim 1, wherein sending the request to the available AI model comprises:
sending the request to an available AI model according to a weighted round-robin schedule of available AI models that correspond to the AI model type identified in the request, wherein weights in the weighted round-robin schedule are proportional to rate limits of the available AI models that correspond to the AI model types identified in the request.

### Claim 7

A processing system, comprising:
one or more memories comprising computer-executable instructions; and
one or more processors configured to execute the computer-executable instructions and cause the processing system to:
periodically send health-status checks to a vendor application programming interface (API);
obtain as output from the vendor API a health-status response identifying an available AI model;
send a request obtained from a client to the available AI model that corresponds to an AI model type identified in the request and is within a rate limit associated with the available AI model; and
send an answer generated by the available AI model to the client.

### Claim 8 (depends on 7)

The processing system of claim 7, wherein the one or more processors are further configured to:
fetch current instances of the AI models in a group of AI models from an AI models database; and
identify available AI models in the group of AI models, type of AI models in the group of AI models, geographical region of the group of AI models, and vendor of the group of AI models.

### Claim 9 (depends on 7)

The processing system of claim 7, wherein to periodically send the health-status check to the vendor API, the one or more processors are configured to cause the processing system to send a health-status check to the vendor API at regular intervals.

### Claim 10 (depends on 7)

The processing system of claim 7, wherein to obtain as output from the vendor API the health-status response identifying the available AI model, the one or more processors are configured to cause the processing system to:
receive the health-status response from the vendor API, the health-status response identifies each AI model in a group of AI models as an available AI model if the AI model is active or an unavailable AI model if the AI model not active;
update an AI model health status data store with information identifying available AI models and unavailable AI models based on the health-status response; and
automatically send an event-status report to stakeholders, the event-status report identifying the available AI models and the unavailable AI models in the group of AI models.

### Claim 11 (depends on 10)

The processing system of claim 10, wherein the one or more processors are further configured to:
for each respective unavailable AI model of the group of AI models:
send health-status checks to the vendor API at shortened intervals until the vendor API sends a health-status response that identifies the respective unavailable AI model as an available AI model,
wherein the shortened intervals have a shorter duration than regular intervals between health-status checks.

### Claim 12 (depends on 7)

The processing system of claim 7, wherein to send the request to the available AI model, the one or more processors are configured to:
send the request to an available AI model according to a weighted round-robin schedule of available AI models that correspond to the AI model type identified in the request, wherein weights in the weighted round-robin schedule are proportional to rate limits of the available AI models that correspond to the AI model types identified in the request.

### Claim 13

An apparatus comprising:
a scheduler application programming interface (API) configured to periodically identify available artificial intelligence (AI) models and unavailable AI models in a group of AI models; and
an AI model API configured to:
receive requests to use particular AI model types sent from a plurality of clients;
distribute the requests to the available AI models that correspond to the AI model types requested by the clients and are within rate limits of the available AI models; and
send answers generated by the available AI models to the plurality of clients in response to the requests.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein the scheduler API comprises:
a time tick processor configured to get current status of the AI models; and
a check endpoint processor configured to send health-status checks to vendor APIs in regular intervals, each vendor API performs a health-status of corresponding AI models.

### Claim 15 (depends on 13)

The apparatus of claim 13, wherein the scheduler API is configured to:
periodically send a health-status check to vendor APIs;
receive a health-status response from the vendor APIs identifying the available AI models and the unavailable AI models;
update an AI model health status data store with information identifying the available AI models and the unavailable AI models; and
automatically send event-status reports via email to stakeholders, each event-status report identifying the available AI models and the unavailable AI models.

### Claim 16 (depends on 13)

The apparatus of claim 13, wherein the scheduler API is configured to send an AI model status report that identifies the available AI models and unavailable AI models to an AI interface API following each health-status check for the available AI models and the unavailable AI models.

### Claim 17 (depends on 13)

The apparatus of claim 13, wherein the scheduler API is configured to, for each of the unavailable AI models, send health-status checks to a vendor API of a respective unavailable AI model at shortened intervals until the respective unavailable AI model is identified as an available AI model by the vendor API.

### Claim 18 (depends on 13)

The apparatus of claim 13, wherein the scheduler API is configured to send an event-status report via email to stakeholders, the event-status report identifying the available AI models and the unavailable AI models.

### Claim 19 (depends on 13)

The apparatus of claim 13, wherein:
the AI model API comprises a load balancer configured to distribute the requests to the available AI models according to a weighted round-robin schedule, and
weights of the weighted round-robin schedule are proportional to rate limits of the available AI models that correspond to the AI model types identified in the requests.

### Claim 20 (depends on 19)

The apparatus of claim 19, wherein the load balancer is configured to:
send a largest number of requests to an available AI model with a largest corresponding rate limit, and
send a smallest number of prompts to an available AI model with a smallest corresponding rate limit.
