Back to results
US 12,706,832 B1Granted
Global endpoints
AssigneeAmazon Technologies, Inc.
InventorsChase Kernan, Christopher Magee Greenwood, Karan Hemant Kothari, Jeffrey Daniel Solinsky
Life of this patent
Estimating the term…
Abstract
Techniques for supporting a token based service are described. In some examples, a token based service such as a foundation model service supports global endpoints. Global endpoints for the token based service are provided with a global quota that is exposed per account, per model. Invocations against a global endpoint consume against the global quota.
Claims20
A computer-implemented method comprising:
- receiving a request from an account to perform an operation using a shared resource at a global endpoint in a first type of region;
- determining a second type of region for a pairing of the account and global endpoint, wherein all calls to the shared resource from any first type of region go through the second type of region;
- calling the determined second type of region to obtain a quota lease for the pairing of the account and global endpoint, wherein a quota lease is a subset of a global quota for the shared resource;
- recording the quota lease in the second type of region;
- performing quota enforcement in the first type of region to determine the request can be serviced using the quota lease;
- calling a third type of region to perform throttling enforcement for the request to determine the request can be serviced;
- generating a routing decision for the request, wherein the routing decision is to route the request to a backend of a plurality of backends;
- routing the request;
- performing the operation;
- and
- providing a result of the operation.
8 more elements
A system comprising:
- a first one or more computing devices in a first type of region to support a foundation model service in a multi-tenant provider network;
- a second one or more computing devices in a second type of region to support a foundation model service in a multi-tenant provider network;
- a third one or more computing devices in a third type of region to support a foundation model service in a multi-tenant provider network;
- a fourth one or more computing devices in a fourth region to support a foundation model service in a multi-tenant provider network, wherein the regions are to cooperate to:
- receive a request from an account to perform an operation using a shared resource at a global endpoint in a first type of region;
- determine a second type of region for determine a second type of region for a pairing of the account and global endpoint, wherein all calls to the shared resource from any first type of region go through the second type of region;
- calling the determined second type of region to obtain a quota lease for the pairing of the account and global endpoint, wherein a quota lease is a subset of a global quota for the shared resource;
- recording the quota lease in the second type of region;
- performing quota enforcement in the first type of region to determine the request can be serviced using the quota lease;
- calling a third type of region to perform throttling enforcement for the request to determine the request can be serviced;
- generating a routing decision for the request, wherein the routing decision is to route the request to a backend of a plurality of backends;
- routing the request;
- performing the operation;
- and
- providing a result of the operation.
12 more elements
A computer-implemented method comprising:
- receiving an inference request from an account for a foundation model at a global endpoint in an originating region;
- determining a root region for a pairing of the account and global endpoint, wherein all calls to the foundation model from any originating region go through the root region;
- calling the determined root region to obtain a quota lease for the pairing of the account and global endpoint, wherein a quota lease is a subset of a global quota for the foundation model;
- recording the quota lease in the root region;
- performing quota enforcement in the originating region to determine the request can be serviced using the quota lease;
- calling a serving region to perform concurrency-based enforcement for the request to determine the request can be serviced;
- generating a routing decision for the request, wherein the routing decision is to route the request to a backend of a plurality of backends;
- routing the request;
- performing the inference using the foundation model;
- and
- providing a result of the inference.
8 more elements
Dependencies are inferred from the claim text.