# US 11,113,717 B2 · Customer engagement platform experimentation framework

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,113,717 B2
- Document ID: US11113717B2
- Status: Granted
- Kind: B2
- Application number: 15/599,968
- Filed: 2017-05-19
- Published: 2018-11-22
- Granted: 2021-09-07
- Earliest claimed: 2017-05-19
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Naganandhini Kohareswaran; Ling-Ya Huang; Einat Hazan; Harish Jayanti; Keith Robert Kinnan; Jayaram N. M. Nanduri; Jagannatha Raju Dantuluri
- Web page: https://patentgrep.com/patent/US11113717B2

## CPC

- G06Q 30/0244: {Optimization}
- G06Q 30/0243: {Comparative campaigns}
- G06Q 30/0255: {based on user history}
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Systems and methods provide a customer engagement platform experimentation framework. Experiments are created to determine which variation of a particular promotion (i.e., an experiment) is most effective. Initially, users are split into buckets that define whether a particular user is in a control group or a treatment group for a particular experiment. Users are assigned into buckets differently for unrelated experiments and similarly for related experiments. This ensures the users are in either the control group or the treatment group for all related experiments. Key performance indicators (KPIs) of users that experienced the experiment are compared with KPIs of users that did not experience the experiment. An assessment of the experiment can then be provided based on the compared KPIs.

## Claims

### Claim 1

A method of creating a customer engagement platform experimentation framework, comprising:
creating, based on selections made by a user at a user interface, an experiment based on an original promotion to determine customer engagement, the experiment including conditions selected by the user;
computing user bucket assignments for the experiment, the user bucket assignments utilizing numberlines to compute the user bucket assignments and group users of the experiment into a control or a treatment group, the numberlines computing the user bucket assignments similarly for related experiments to ensure the users are in either the control or the treatment group for all the related experiments;
creating experiment bookings for the experiment in accordance with the user bucket assignments;
detecting parameters at runtime during a client content request, the parameters including operating system version or screen resolution;
and
upon the detected parameters being satisfied, delivering the experiment to the users in accordance with the conditions and the experiment bookings.

### Claim 2 (depends on 1)

The method of claim 1, wherein the numberlines compute the user bucket assignments differently for unrelated experiments.

### Claim 3 (depends on 1)

The method of claim 1, further comprising assigning different flight percentages for various segments within the treatment group in accordance with the conditions.

### Claim 4 (depends on 1)

The method of claim 1, further comprising comparing key performance indicators (KPIs) of the users that experienced the experiment with users that did not experience the experiment.

### Claim 5 (depends on 4)

The method of claim 4, further comprising providing an assessment of the experiment based on the compared KPIs.

### Claim 6 (depends on 1)

The method of claim 1, further comprising implementing a control group by omitting a set of buckets of the users in accordance with a size of the control group in all experiments associated with a particular numberline.

### Claim 7 (depends on 1)

The method of claim 1, wherein the experiment is revealed to a configurable percentage of users initially.

### Claim 8 (depends on 6)

The method of claim 6, ramping up or down the experiment based upon an assessment of an effectiveness of the experiment.

### Claim 9 (depends on 1)

The method of claim 1, wherein exactly one parameter within the original promotion can be varied.

### Claim 10 (depends on 8)

The method of claim 8, wherein the experiment can test more than one value for this parameter.

### Claim 11 (depends on 9)

The method of claim 9, wherein the treatment group includes an exposure percentage that determines a percentage of the users that will experience a particular variant value of the experiment.

### Claim 12

A computer-readable storage media comprising computer-executable instructions, that when executed by a computer, causes the computer to perform operations to facilitate creating a customer engagement platform experimentation framework, the operations comprising:
creating, based on selections made by a user at a user interface, an experiment based on an original promotion to determine customer engagement, the experiment including conditions selected by the user;
utilizing numberlines, computing user bucket assignments for the experiment that group the users of the experiment into a control or a treatment group, the numberlines computing the user bucket assignments differently for unrelated experiments and similarly for related experiments to ensure the users are in either the control or the treatment group for all the related experiments;
creating experiment bookings for the experiment in accordance with the user bucket assignments;
detecting parameters at runtime during a client content request, the parameters including operating system version or screen resolution;
upon the detected parameters being satisfied, delivering the experiment to the users in accordance with the conditions and the experiment bookings;
comparing key performance indicators (KPIs) of users that experienced the experiment with KPIs of users that did not experience the experiment; and
providing an assessment of the experiment based on the compared KPIs.

### Claim 13 (depends on 12)

The media of claim 12, further comprising creating experiment bookings for the experiment in accordance with the bucket assignments.

### Claim 14 (depends on 13)

The media of claim 13, further comprising delivering the experiment to user clients in accordance with the conditions and the experiment bookings.

### Claim 15 (depends on 12)

The media of claim 12, wherein exactly one parameter within the original promotion can be varied.

### Claim 16 (depends on 15)

The media of claim 15, wherein the experiment can test more than one value for this parameter.

### Claim 17 (depends on 16)

The media of claim 16, wherein the treatment group includes an exposure percentage that determines the percentage of users that will experience a particular variant value of the experiment.

### Claim 18

A computerized system:
a processor; and
a computer storage medium storing computer-useable instructions that, when used by the processor, cause the processor to:
enable setup and management of experiments to determine customer engagement, the setup and management including conditions;
store and manage the experiments and numberline metadata;
compute user bucket assignments for each experiment of the experiments in accordance with the numberline metadata, wherein the numberline metadata is utilized to compute the user bucket assignments similarly for related experiments to ensure the users are in either the control or the treatment group for all the related experiments;
create experiment bookings for each experiment of the experiments in accordance with the user bucket assignments;
detect parameters at runtime during a client content request, the parameters including operating system version or screen resolution;
and
upon the detected parameters being satisfied, deliver the experiment to the users in accordance with the conditions and the experiment bookings.