# US 12,764,034 B1 · Secure multi-model hosting and experimentation in a cloud provider network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,764,034 B1
- Document ID: US12764034B1
- Status: Granted
- Kind: B1
- Application number: 18/797,155
- Filed: 2024-08-07
- Granted: 2026-10-13
- Earliest claimed: 2024-08-07
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: Karthik Ravindra Rao; Sai Srinivas Somarouthu; Kenneth John Sanchez Venegas; Vladimir Mitrovic; Himani Amrish Shah; Matthew Lee; Rama Krishna Sandeep Pokkunuri; Vamshidhar Dantu Krishnamurthy; Alice Aizawa; Srinivas Iragavarapu; Qiang Yu; Sathish Arumugam Selvaraj
- Web page: https://patentgrep.com/patent/US12764034B1

## CPC

- H04L 41/16: using machine learning or artificial intelligence
- H04L 41/14: Network analysis or design
- H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Techniques for secure multi-model hosting and experimentation in a cloud provider network are described. A request to perform an action involving one of multiple machine learning (ML) models is received at a cloud provider network. A user associated with the request can be consistently mapped to a particular experience from multiple experiences of an experiment. Based on this experience, a request to use a ML model can be sent by a data plane component that includes a path value associated with the experience. The request can be sent from the data plane component to a private connectivity endpoint, which securely provides the request into another virtual private cloud including the ML models without the request transiting the public internet. The request can be routed via path-based routing and the included path value to a ML model providing the experience.

## Claims

### Claim 1

A computer-implemented method comprising:
receiving, at a service of a cloud provider network, a request from a user to generate source code;
selecting, based on a user identifier of the user, an experience associated with the user, wherein the selecting comprises:
generating a bucket identifier via use of a deterministic function, and identifying the experience based on use of the bucket identifier;
generating an application layer resource identifier comprising a path, wherein the path includes a path component that corresponds to the identified experience;
transmitting a request to perform an inference to a private connectivity endpoint, the request including the application layer resource identifier comprising a uniform resource locator (URL) address;
routing the request to a first generative machine learning (ML) model, of multiple generative ML models, based on the path component in the URL, wherein:
the routing is performed by an application layer load balancer that uses application-layer routing rules, including path-based routing rules, to select among the multiple generative ML models,
the private connectivity endpoint provides access to each of the multiple generative ML models and different path components in the URLs route to different generative ML models, and
the request is routed entirely within the cloud provider network and does not transit the public internet;
generating an output via use of the first generative ML model, the output comprising the source code; and
transmitting the output to a computing device of the user to be presented.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
receiving, at the cloud provider network, user-specified dial-up configuration data for an experiment, the dial-up configuration data indicating how to consistently split users between at least a first experience and a second experience, wherein the first experience is associated with the first ML generative model and the second experience is associated with a second ML generative model,
wherein the selected experience comprises the first experience, and
wherein the selecting of the experience is based on use of at least some of the dial-up configuration data.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the transmitted request is sent within a cloud private network, from a data plane component operating in the cloud private network, to a private connectivity endpoint that is also in the cloud private network, wherein the private connectivity endpoint provides connectivity to a second cloud private network without transiting the public internet, wherein the first generative ML model operates within the second cloud private network.

### Claim 4

A computer-implemented method comprising:
receiving, at a service of a cloud provider network, a request originated on behalf of a user to perform an action involving generating an output based on input from the request, the action involving use of at least one of multiple machine learning (ML) models available within the cloud provider network;
selecting, based on a user identifier of the user, an experience associated with the user;
generating an application layer resource identifier comprising a path, wherein the path includes a path component that corresponds to the selected experience;
transmitting a request to perform an inference to a private connectivity endpoint, the request including the application layer resource identifier;
routing the request, by an application layer load balancer using application-layer routing rules, including path-based routing rules, to a first ML model of the multiple ML models based on the path component; and
generating the output via use of the first ML model.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein selecting the experience associated with the user comprises utilizing a deterministic algorithm to generate a bucket identifier.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, wherein the deterministic algorithm comprises a consistent hashing algorithm, and wherein utilizing the deterministic algorithm comprises utilizing at least the user identifier as an input to the consistent hashing algorithm as part of generating the bucket identifier.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein selecting the experience further comprises identifying, based on the bucket identifier and further based on an identifier of an experiment associated with the action, the first ML model or a component for the path.

### Claim 8 (depends on 4)

The computer-implemented method of claim 4, further comprising:
receiving, at the cloud provider network, user-specified dial-up configuration data for an experiment, the dial-up configuration data indicating how to consistently split users between at least a first experience and a second experience, wherein the first experience is associated with the first ML model and the second experience is associated with a second ML model,
wherein the selected experience comprises the first experience, and
wherein the selecting of the experience is based on at least some of the dial-up configuration data.

### Claim 9 (depends on 4)

The computer-implemented method of claim 4, wherein the transmitted request is sent to the private connectivity endpoint within a cloud private network, from a data plane component operating in the cloud private network, to a private connectivity endpoint that is also in the cloud private network, wherein the private connectivity endpoint provides connectivity to a second cloud private network without transiting the public internet, wherein the first ML model operates within the second cloud private network.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein routing the request includes processing the request via the application layer load balancer using application-layer routing rules, including path-based routing rules, wherein the application layer load balancer operates within the second cloud private network.

### Claim 11 (depends on 4)

The computer-implemented method of claim 4, wherein the experience is part of a first experiment involving the multiple ML models, and wherein the method further comprises:
selecting, based on the user identifier of the user, a second experience associated with the user, wherein the second experience is part of a second experiment that is different than the experiment.

### Claim 12 (depends on 11)

The computer-implemented method of claim 11, further comprising:
transmitting, to a client device associated with the user, a command indicating that the client device is to enable a functionality associated with the second experience.

### Claim 13 (depends on 4)

The computer-implemented method of claim 4, further comprising:
generating a first set of metrics associated with activity of at least the user that indicate that the activity corresponds to the experience; and
generating a second set of metrics associated with activity of at least another user that indicate that the activity corresponds to a second experience,
wherein the experience and the second experience are part of a same experiment.

### Claim 14 (depends on 4)

The computer-implemented method of claim 4, wherein the user is further associated, via a same bucket identifier, with a second experience of a second experiment.

### Claim 15 (depends on 14)

The computer-implemented method of claim 14, wherein the experience involves a server-side functionality, the second experience involves a client-side functionality, and both the experience and the second experience are concurrently active.

### Claim 16

A system comprising:
a first one or more computing devices to implement a control plane of a service in a cloud provider network, the control plane including instructions that upon execution cause the control plane to:
receive user-specified experiment configuration data; and
configure one or more components of a data plane based on the user-specified experiment configuration data;
a second one or more computing devices to implement the data plane of the service in the cloud provider network, the data plane including instructions that upon execution cause the data plane to:
receive a request originated on behalf of a user to perform an action involving generation of an output based on input from the request, the action involving use of at least one of multiple machine learning (ML) models available within the cloud provider network;
select, based on a user identifier of the user, an experience associated with the user;
generate an application layer resource identifier comprising a path, wherein the path includes a path component corresponding to the selected experience;
transmit a request to perform an inference to a private connectivity endpoint, the request including the application layer resource identifier;
route the request, via an application layer load balancer using application-layer routing rules, including path-based routing rules, to a first ML model of the multiple ML models based on the path component; and
generate the output via use of the first ML model.

### Claim 17 (depends on 16)

The system of claim 16, wherein to select the experience associated with the user, the data plane is at least to utilize at least the user identifier as an input to a consistent hashing algorithm as part of generating a bucket identifier.

### Claim 18 (depends on 16)

The system of claim 16, wherein:
the user-specified experiment configuration data comprises user-specified dial-up configuration data for an experiment, the dial-up configuration data indicating how to consistently split users between at least a first experience and a second experience, wherein the first experience is associated with the first ML model and the second experience is associated with a second ML model;
the selected experience comprises the first experience; and
the selection of the experience is based on at least some of the dial-up configuration data.

### Claim 19 (depends on 16)

The system of claim 16, wherein the transmitted request is sent to the private connectivity endpoint within a cloud private network, from a data plane component operating in the cloud private network, to a private connectivity endpoint that is also in the cloud private network, wherein the private connectivity endpoint provides connectivity to a second cloud private network without transiting the public internet, wherein the first ML model operates within the second cloud private network.

### Claim 20 (depends on 19)

The system of claim 19, wherein to route the request the data plane is at least to process the request via the application layer load balancer using application-layer routing rules, including path-based routing rules, wherein the application layer load balancer operates within the second cloud private network.