US 11,537,378 B2Granted

Selecting a sample set of cloud computing resources for A/B testing of a software deployment

AssigneeMicrosoft Technology Licensing, LLC

InventorsMathieu Baptiste Demarne, Miso Cilimdzic

Life of this patent

Estimating the term…

Loading patent…

Abstract

A sample set of cloud computing resources is dynamically selected for testing a software deployment. Telemetry data associated with the resources and customer support data associated with customers that utilize the resources are obtained. A subset of the customers is selected based on the customer support data, and a candidate subset of the resources is selected based on the selected subset of customers and the telemetry data. Criteria for the selection is based on usage patterns and is configurable. Resources of customers with special support agreements, and customers previously selected, may be excluded from the candidate subset. The sample set of cloud computing resources may be randomly selected from the candidate subset. Software is deployed to the sample set as a B resource group and tested for issues with an A resource group to determine whether to proceed to full deployment, roll back the deployment, and/or retest the software.

Claims20

Dependent claims

A system implemented in a computer for dynamically selecting from among a plurality of cloud computing resources of a cloud computing provider a sample set of cloud computing resources for testing a software deployment, the plurality of cloud computing resources being deployed on behalf of a plurality of customers of the cloud computing provider to support a corresponding plurality of cloud computing services, the system comprising:

one or more processors;
and
one or more memory devices that store program code to be executed by the one or more processors, the program code comprising:
a selection manager configured to:
retrieve stored telemetry data comprising measurements collected from the plurality of cloud computing resources of the cloud computing provider for use in selecting cloud computing resources for participating in a software deployment test of particular software executing on the sample set of cloud computing resources:
retrieve stored customer support data comprising user input cloud resource issues associated with each of the plurality of customers, wherein each of the plurality of customers is associated with one or more of the plurality of cloud computing resources;
select a subset of the plurality of customers of the cloud computing provider for inclusion in the software deployment test based at least on the customer support data;
select a candidate subset of the plurality of cloud computing resources for participating in the software deployment test based at least on the selected subset of the plurality of customers, aspects of the particular software, and the telemetry data, wherein the telemetry data of each resource of the candidate subset indicates workload activity sufficient for testing the aspects of the particular software;
and
randomly select the sample set of cloud computing resources for participating in the software deployment test from among the candidate subset;
and
a software deployment manager configured to:
deploy the particular software in the sample set of cloud computing resources;
and
execute the deployment software test in the sample set of cloud computing resources.

12 more elements

A computer implemented method for dynamically selecting from among a plurality of cloud computing resources of a cloud computing provider a sample set of cloud computing resources for testing a software deployment, the plurality of cloud computing resources being deployed on behalf of a plurality of customers of the cloud computing provider to support a corresponding plurality of cloud computing services, the method comprising:

retrieving stored telemetry data comprising measurements collected from the plurality of cloud computing resources of the cloud computing provider for use in selecting cloud computing resources for participating in a software deployment test of particular software executing on the sample set of cloud computing resources;
retrieving stored customer support data comprising user input cloud resource issues associated with each of the plurality of customers, wherein each of the plurality of customers is associated with one or more of the plurality of cloud computing resources;
selecting a subset of the plurality of customers of the cloud computing provider for inclusion in the software deployment test based at least on the customer support data;
selecting a candidate subset of the plurality of cloud computing resources for participating in the software deployment test based at least on the selected subset of the plurality of customers, aspects of the particular software, and the telemetry data, wherein the telemetry data of each resource of the candidate subset indicates workload activity sufficient for testing the aspects of the particular software;
and
selecting the sample set of cloud computing resources for participating in the software deployment test from among the candidate subset;
deploying the particular software in the sample set of cloud computing resources;
and
executing the deployment software test in the sample set of cloud computing resources.

6 more elements

A computer-readable storage medium having program code recorded thereon that when executed by at least one processor causes the at least one processor to perform a method for dynamically selecting from among a plurality of cloud computing resources of a cloud computing provider a sample set of cloud computing resources for testing a software deployment, the plurality of cloud computing resources being deployed on behalf of a plurality of customers of the cloud computing provider to support a corresponding plurality of cloud computing services, the method comprising:

retrieving stored telemetry data comprising measurements collected from the plurality of cloud computing resources of the cloud computing provider for use in selecting cloud computing resources for participating in a software deployment test of particular software executing on the sample set of cloud computing resources;
retrieving stored customer support data comprising user input cloud resource issues associated with each of the plurality of customers, wherein each of the plurality of customers is associated with one or more of the plurality of cloud computing resources;
selecting a subset of the plurality of customers of the cloud computing provider for inclusion in the software deployment test based at least on the customer support data;
selecting a candidate subset of the plurality of cloud computing resources for participating in the software deployment test based at least on the selected subset of the plurality of customers, aspects of the particular software, and the telemetry data, wherein the telemetry data of each resource of the candidate subset indicates workload activity sufficient for testing the aspects of the particular software;
and
randomly selecting the sample set of cloud computing resources for participating in the software deployment test from among the candidate subset;
deploying the particular software in the sample set of cloud computing resources:
and
executing the deployment software test in the sample set of cloud computing resources.

6 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/193,399Filed 5 Mar 2021
Published: US 2022/0283793 A18 Sep 2022
Granted: 27 Dec 2022
Term ends: Not estimatedThe source records don’t support an estimate.

Classification8 codes

G06F 8/60
Arrangements for software engineering: software deployment
G06F 11/302
Monitoring arrangements specially adapted to the computing system or computing system component being monitored: where the computing system…
G06F 9/5027
To service a request: the resource being a machine, e.g. CPUs, Servers, Terminals
G06F 11/3409
Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of…:…