# US 2025/0147871 A1 · Imbalance detection in online experiments

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0147871 A1
- Document ID: US20250147871A1
- Status: Published application
- Kind: A1
- Application number: 19/015,168
- Filed: 2025-01-09
- Published: 2025-05-08
- Earliest claimed: 2023-02-27
- Applicants: Etsy, Inc.
- Inventors: Stephane Shao; Lyubov Ignatovsky; Julie Beckley
- Web page: https://patentgrep.com/patent/US20250147871A1

## CPC

- G06F 9/5083: {Techniques for rebalancing the load in a distributed system}
- G06F 11/3696: {Methods or tools to render software testable}
- G06F 8/71: Version control (security arrangements therefor G06F21/57); Configuration management
- G06F 11/3688: {for test execution, e.g. scheduling of test suites}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods, systems, and apparatus, including computer programs encoded on computer-storage media, for imbalance detection in online experiments. In some implementations, a method includes obtaining user information indicating a first set of devices assigned a first version of a service and a second set of devices assigned a second version of the service in a multivariate testing framework; generating alternative samplings of devices; generating a threshold for detecting imbalance using the alternative samplings of devices, a generated expected first set of devices, and a generated expected second set of devices; detecting an imbalance using the obtained user information indicating the first set of devices, the second set of devices, and the generated threshold; and implementing a corrective action to mitigate the detected imbalance in the multivariate testing framework.

## Claims

### Claim at position 1

1-20. (canceled)

### Claim 21

A method comprising:
receiving user information indicating a first set of devices assigned to a first version of a service and a second set of devices assigned to a second version of the service in a multivariate testing framework;
generating an imbalance threshold using a third set of devices expected to be assigned to the first version of the service and a fourth set of devices expected to be assigned to the second version of the service, wherein the imbalance threshold indicates whether there is an imbalance in the multivariate testing framework;
determining the imbalance in the multivariate testing framework using the user information and the imbalance threshold; and
implementing a corrective action to mitigate the determined imbalance in the multivariate testing framework.

### Claim 22 (depends on 21)

The method of claim 21, wherein the user information is received from a computer system that assigns the first version of the service to the first set of devices and the second version of the service to the second set of devices.

### Claim 23 (depends on 21)

The method of claim 21, further comprising:
generating alternative samplings of devices including the third set of devices and the fourth set of devices that are assigned to the first version and the second version of the service, respectively.

### Claim 24 (depends on 23)

The method of claim 23, wherein generating the alternative samplings of devices comprises:
performing one or more permutations on values representing segments or variants of the user information indicating the first set of devices assigned the first version of the service and the second set of devices assigned the second version of the service.

### Claim 25 (depends on 23)

The method of claim 23, wherein generating the alternative samplings of devices comprises:
sampling from a distribution indicating a numerical range of types of devices assigned to the first version or the second version of the service, where the types of devices that receive either the first version or the second version of the service is independent from the types of devices.

### Claim 26 (depends on 21)

The method of claim 21, comprising:
generating the third set of devices expected to be assigned to the first version of the service and the fourth set of devices expected to be assigned to the second version of the service by generating values representing at least (i) a number of devices of a first segment assigned to the first version of the service, (ii) a number of devices of a second segment assigned to the second version of the service, (iii) a number of devices of the first segment assigned to the second version of the service, and (iv) a number of devices of the second segment assigned to the first version of the service.

### Claim 27 (depends on 21)

The method of claim 21, further comprising:
generating the third set of devices expected to be assigned to the first version of the service and the fourth set of devices expected to be assigned to the second version of the service.

### Claim 28 (depends on 27)

The method of claim 27, wherein generating the third set of devices expected to be assigned to the first version of the service and the fourth set of devices expected to be assigned to the second version of the service comprises:
generating (i) one or more values indicating a number of devices of a specific type expected to be assigned the first version and (ii) one or more values indicating a number of devices of a specific type expected to be assigned the second version of the service.

### Claim 29 (depends on 28)

The method of claim 28, wherein the devices of the specific type are devices operating in a specific region, operating with a specific language preference, operating on a specific hardware, or operating on a specific software.

### Claim 30 (depends on 21)

The method of claim 21, wherein generating the imbalance threshold for determining the imbalance using the user information comprises:
generating one or more difference values indicating a difference between (i) a sample of alternative samplings of devices assigned to the first version of the service and the second version, respectively, and (ii) the third set of devices expected to be assigned to the first version of the service and the fourth set of devices expected to be assigned to the second version of the service.

### Claim 31 (depends on 21)

The method of claim 21, wherein generating the imbalance threshold comprises selecting an alpha-quantile of a difference distribution, wherein the difference distribution is computed by comparing (i) a sample of alternative samplings of devices assigned to the first version of the service and the second version, respectively, and (ii) the third set of devices expected to be assigned to the first version of the service and the fourth set of devices expected to be assigned to the second version of the service.

### Claim 32 (depends on 21)

The method of claim 21, further comprising:
generating one or more difference values between (i) the user information, indicating the first set of devices and the second set of devices, and (ii) the third set of devices expected to be assigned to the first version of the service and the fourth set of devices expected to be assigned to the second version of the service; and
wherein determining the imbalance using the user information and the imbalance threshold comprises determining that the one or more difference values generate a difference value that is equal to or greater than the imbalance threshold.

### Claim 33 (depends on 21)

The method of claim 21, wherein implementing the corrective action to mitigate the determined imbalance in the multivariate testing framework comprises:
generating a notification for a testing device; and
transmitting the notification to the testing device.

### Claim 34 (depends on 33)

The method of claim 33, wherein the notification is configured to halt one or more data processing processes on the testing device.

### Claim 35 (depends on 21)

The method of claim 21, wherein implementing the corrective action to mitigate the determined imbalance in the multivariate testing framework comprises:
generating a signal configured to assign additional devices to the first version or the second version to reduce the determined imbalance.

### Claim 36 (depends on 35)

The method of claim 35, wherein the additional devices comprise a device of a particular type represented by one or more device or user features associated with the determined imbalance.

### Claim 37

A system comprising one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving user information indicating a first set of devices assigned to a first version of a service and a second set of devices assigned to a second version of the service in a multivariate testing framework;
generating an imbalance threshold using a third set of devices expected to be assigned to the first version of the service and a fourth set of devices expected to be assigned to the second version of the service, wherein the imbalance threshold indicates whether there is an imbalance in the multivariate testing framework;
determining the imbalance in the multivariate testing framework using the user information and the imbalance threshold; and
implementing a corrective action to mitigate the determined imbalance in the multivariate testing framework.

### Claim 38 (depends on 37)

The system of claim 37, wherein the user information is received from a computer system that assigns the first version of the service to the first set of devices and the second version of the service to the second set of devices.

### Claim 39 (depends on 37)

The system of claim 37, wherein the operations comprise:
generating alternative samplings of devices including the third set of devices and the fourth set of devices that are assigned to the first version and the second version of the service, respectively.

### Claim 40

One or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
receiving user information indicating a first set of devices assigned to a first version of a service and a second set of devices assigned to a second version of the service in a multivariate testing framework;
generating an imbalance threshold using a third set of devices expected to be assigned to the first version of the service and a fourth set of devices expected to be assigned to the second version of the service, wherein the imbalance threshold indicates whether there is an imbalance in the multivariate testing framework;
determining the imbalance in the multivariate testing framework using the user information and the imbalance threshold; and
implementing a corrective action to mitigate the determined imbalance in the multivariate testing framework.
