# US 12,222,852 B2 · Imbalance detection in online experiments > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,222,852 B2 - Document ID: US12222852B2 - Status: Granted - Kind: B2 - Application number: 18/114,773 - Filed: 2023-02-27 - Published: 2023-08-31 - Granted: 2025-02-11 - Earliest claimed: 2022-02-25 - Assignees: Etsy, Inc. - Applicants: Etsy, Inc. - Inventors: Stephane Shao; Lyubov Ignatovsky; Julie Beckley - Web page: https://patentgrep.com/patent/US12222852B2 ## CPC - G06F 9/5083: {Techniques for rebalancing the load in a distributed system} - G06F 11/3696: {Methods or tools to render software testable} - G06F 8/71: Version control (security arrangements therefor G06F21/57); Configuration management - G06F 11/3688: {for test execution, e.g. scheduling of test suites} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Methods, systems, and apparatus, including computer programs encoded on computer-storage media, for imbalance detection in online experiments. In some implementations, a method includes obtaining user information indicating a first set of devices assigned a first version of a service and a second set of devices assigned a second version of the service in a multivariate testing framework; generating alternative samplings of devices; generating a threshold for detecting imbalance using the alternative samplings of devices, a generated expected first set of devices, and a generated expected second set of devices; detecting an imbalance using the obtained user information indicating the first set of devices, the second set of devices, and the generated threshold; and implementing a corrective action to mitigate the detected imbalance in the multivariate testing framework. ## Claims ### Claim 1 A method comprising: obtaining user information indicating a first set of devices assigned a first version of a service and a second set of devices assigned a second version of the service in a multivariate testing framework; generating alternative samplings of devices assigned to the first version and the second version of the service; generating an expected first set of devices assigned to the first version of the service and an expected second set of devices assigned to the second version of the service; generating a threshold for detecting imbalance using the alternative samplings of devices, the expected first set of devices, and the expected second set of devices; detecting an imbalance using the obtained user information indicating the first set of devices, the second set of devices, and the generated threshold; and implementing a corrective action to mitigate the detected imbalance in the multivariate testing framework. ### Claim 2 (depends on 1) The method of claim 1, wherein the user information is obtained from a computer system that provides the first version of the service and the second version of the service to the first set of devices and the second set of devices. ### Claim 3 (depends on 1) The method of claim 1, wherein generating the alternative samplings of devices assigned to the first version and the second version of the service comprise: performing one or more permutations on values representing segments or variants of the user information indicating the first set of devices assigned the first version of the service and the second set of devices assigned the second version of the service. ### Claim 4 (depends on 1) The method of claim 1, wherein generating the alternative samplings of devices assigned the first version and the second version of the service comprise: sampling from a distribution indicating a numerical range of types of devices assigned to the first version or the second version of the service where the types of devices that receive either the first version or the second version of the service is independent from the types of devices. ### Claim 5 (depends on 1) The method of claim 1, wherein generating the expected first set of devices assigned to the first version of the service and the expected second set of devices assigned to the second version of the service comprises: generating one or more values indicating a number of devices of a specific type that receive the first version and the second version of the service. ### Claim 6 (depends on 5) The method of claim 5, wherein the devices of the specific type are devices operating in a specific region, operating with a specific language preference, operating on a specific hardware, or operating on a specific software. ### Claim 7 (depends on 1) The method of claim 1, wherein generating the threshold for detecting the imbalance using the alternative samplings of devices, the expected first set of devices, and the expected second set of devices comprises: generating one or more difference values indicating a difference between a sample of the alternative samplings of devices and a sample of the expected first set of devices and the expected second set of devices. ### Claim 8 (depends on 1) The method of claim 1, wherein generating the threshold for detecting the imbalance using the alternative samplings of devices, the expected first set of devices, and the expected second set of devices comprises using obtained data indicating the threshold for detecting the imbalance to determine a quantile of a distribution created using the alternative samplings of devices, the expected first set of devices, and the expected second set of devices as the threshold for detecting the imbalance. ### Claim 9 (depends on 1) The method of claim 1, comprising: generating one or more difference values between the obtained user information, indicating the first set of devices and the second set of devices, and the expected first and second set of devices; and wherein detecting the imbalance using the obtained user information indicating the first set of devices, the second set of devices, and the generated threshold comprises determining that the one or more difference values generate a cumulative difference value that is equal to or greater than a threshold difference value generated using one or more difference values generated between the alternative samplings of devices and the expected first and second set of devices. ### Claim 10 (depends on 1) The method of claim 1, wherein implementing the corrective action to mitigate the detected imbalance in the multivariate testing framework comprises: generating a notification for a testing device; and transmitting the notification to the testing device. ### Claim 11 (depends on 10) The method of claim 10, wherein the notification is configured to halt one or more data processing processes on the testing device. ### Claim 12 (depends on 1) The method of claim 1, wherein implementing the corrective action to mitigate the detected imbalance in the multivariate testing framework comprises: generating a signal configured to assign additional devices to the first version or the second version to reduce the detected imbalance. ### Claim 13 (depends on 12) The method of claim 12, wherein the additional devices comprise a device of a particular type represented by one or more device or user features associated with the detected imbalance. ### Claim 14 A non-transitory computer-readable medium storing one or more instructions executable by a computer system to perform operations comprising: obtaining user information indicating a first set of devices assigned a first version of a service and a second set of devices assigned a second version of the service in a multivariate testing framework; generating alternative samplings of devices assigned to the first version and the second version of the service; generating an expected first set of devices assigned to the first version of the service and an expected second set of devices assigned to the second version of the service; generating a threshold for detecting imbalance using the alternative samplings of devices, the expected first set of devices, and the expected second set of devices; detecting an imbalance using the obtained user information indicating the first set of devices, the second set of devices, and the generated threshold; and implementing a corrective action to mitigate the detected imbalance in the multivariate testing framework. ### Claim 15 (depends on 14) The medium of claim 14, wherein the user information is obtained from a computer system that provides the first version of the service and the second version of the service to the first set of devices and the second set of devices. ### Claim 16 (depends on 14) The medium of claim 14, wherein generating the alternative samplings of devices assigned to the first version and the second version of the service comprise: performing one or more permutations on values representing segments or variants of the user information indicating the first set of devices assigned the first version of the service and the second set of devices assigned the second version of the service. ### Claim 17 (depends on 14) The medium of claim 14, wherein generating the alternative samplings of devices assigned the first version and the second version of the service comprise: sampling from a distribution indicating a numerical range of types of devices assigned to the first version or the second version of the service where the types of devices that receive either the first version or the second version of the service is independent from the types of devices. ### Claim 18 (depends on 14) The medium of claim 14, wherein generating the expected first set of devices assigned to the first version of the service and the expected second set of devices assigned to the second version of the service comprises: generating one or more values indicating a number of devices of a specific type that receive the first version and the second version of the service. ### Claim 19 (depends on 18) The medium of claim 18, wherein the devices of the specific type are devices operating in a specific region, operating with a specific language preference, operating on a specific hardware, or operating on a specific software. ### Claim 20 A system, comprising: one or more processors; and machine-readable media interoperably coupled with the one or more processors and storing one or more instructions that, when executed by the one or more processors, perform operations comprising: obtaining user information indicating a first set of devices assigned a first version of a service and a second set of devices assigned a second version of the service in a multivariate testing framework; generating alternative samplings of devices assigned to the first version and the second version of the service; generating an expected first set of devices assigned to the first version of the service and an expected second set of devices assigned to the second version of the service; generating a threshold for detecting imbalance using the alternative samplings of devices, the expected first set of devices, and the expected second set of devices; detecting an imbalance using the obtained user information indicating the first set of devices, the second set of devices, and the generated threshold; and implementing a corrective action to mitigate the detected imbalance in the multivariate testing framework.