Secure multi-model hosting and experimentation in a cloud provider network

A computer-implemented method comprising:

• receiving, at a service of a cloud provider network, a request originated on behalf of a user to perform an action involving generating an output based on input from the request, the action involving use of at least one of multiple machine learning (ML) models available within the cloud provider network;
• selecting, based on a user identifier of the user, an experience associated with the user;
• generating an application layer resource identifier comprising a path, wherein the path includes a path component that corresponds to the selected experience;
• transmitting a request to perform an inference to a private connectivity endpoint, the request including the application layer resource identifier;
• routing the request, by an application layer load balancer using application-layer routing rules, including path-based routing rules, to a first ML model of the multiple ML models based on the path component;
• and
• generating the output via use of the first ML model.

A system comprising:

• a first one or more computing devices to implement a control plane of a service in a cloud provider network, the control plane including instructions that upon execution cause the control plane to:
• receive user-specified experiment configuration data;
• and
• configure one or more components of a data plane based on the user-specified experiment configuration data;
• a second one or more computing devices to implement the data plane of the service in the cloud provider network, the data plane including instructions that upon execution cause the data plane to:
• receive a request originated on behalf of a user to perform an action involving generation of an output based on input from the request, the action involving use of at least one of multiple machine learning (ML) models available within the cloud provider network;
• select, based on a user identifier of the user, an experience associated with the user;
• generate an application layer resource identifier comprising a path, wherein the path includes a path component corresponding to the selected experience;
• transmit a request to perform an inference to a private connectivity endpoint, the request including the application layer resource identifier;
• route the request, via an application layer load balancer using application-layer routing rules, including path-based routing rules, to a first ML model of the multiple ML models based on the path component;
• and
• generate the output via use of the first ML model.