# US 2026/0065302 A1 · Computer-implemented Method for Global Holdout Testing and Related Systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0065302 A1
- Document ID: US20260065302A1
- Status: Published application
- Kind: A1
- Application number: 19/285,676
- Filed: 2025-07-30
- Published: 2026-03-05
- Earliest claimed: 2024-08-27
- Assignees: Klaviyo, Inc
- Applicants: Klaviyo, Inc.
- Inventors: Justin Xu; Carola Leiva; Jack Cosgrove
- Web page: https://patentgrep.com/patent/US20260065302A1

## CPC

- G06Q 30/020112: {Analysing events leading to purchases, e.g. "purchase journey" or attribution}
- G06Q 30/0201: Market modelling; Market analysis; Collecting market data
- G06Q 30/0204: Market segmentation
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

A computer-implemented method for computing a lift value for multiple types of electronic messaging strategies is based on global holdout group testing. A set of preprocessing rules and an optional classifier are applied to identify the total number of valid profile base objects available for testing. The total number of valid profile base objects are assigned to the holdout group and the non-holdout group. Raw events are joined with the group events. An analytics engine computes the conversions rates such as lift value and win probability.

## Claims

### Claim 1

A computer-implemented method for computing a lift value for global holdout testing, wherein the global holdout testing measures conversions for multiple types of electronic messages and sending strategies, the method comprising:
receiving a plurality of testing parameters comprising holdout percentage for a holdout group and test duration;
selecting electronic message sending strategies to override from the holdout group;
retrieving, from a profile base object database, valid profile base objects;
preprocessing the valid profile base objects based on a set of exclusion rules;
assigning, by a server, profile base objects to the holdout group and profile base objects to the non-holdout group;
publishing asynchronously, in batches, assigned membership event base objects corresponding to the holdout group and the non-holdout group;
joining, by profile, the assigned membership event base objects and unassigned raw event base objects received from a raw event pipeline to create an updated set of event base objects for the holdout group and for the non-holdout group;
calculating total counts of the event base objects in the holdout group and total counts of the event base objects in the non-holdout group;
calculating conversion counts of conversion events for the event base objects in the holdout group and conversion counts for conversion events for the event base objects in the non-holdout group; and
computing, by the server, at least one conversion rate based on the test duration, conversion counts, and total counts.

### Claim 2 (depends on 1)

The method of claim 1, wherein the conversion count is based on a statistic type selected from the group comprising purchases, open carts, and checkouts started.

### Claim 3 (depends on 1)

The method of claim 1, wherein the messaging strategy types include one-time manual message sends and automatic sends based on trigger events.

### Claim 4 (depends on 1)

The method of claim 1, wherein the selecting step is performed by a selector model operable to classify candidates to override from the holdout group.

### Claim 5 (depends on 1)

The method of claim 1, wherein the assigning is performed by a hashing algorithm.

### Claim 6 (depends on 1)

The method of claim 1, wherein the assigning is performed in real-time within volatile memory structures without committing data to persistent storage mediums.

### Claim 7 (depends on 1)

The method of claim 1, further comprising computing a win probability.

### Claim 8 (depends on 7)

The method of claim 7, wherein the win probability is based on the Bayes' formula.

### Claim 9 (depends on 1)

The method of claim 1, wherein the selecting step comprises identifying messages automatically triggered based on a sub-user detected behavior.

### Claim 10 (depends on 1)

The method of claim 1, wherein the exclusion rules are based on excluding profile base objects from the holdout group and are based on channel permissions and optionally, whether profile base object is a VIP.

### Claim 11

A system for computing lift for global holdout testing, wherein the global holdout testing measures conversions for multiple types of electronic messages and sending strategies, the system comprising:
a raw event database operable to manage raw event base objects arising from sub-user behaviors;
a profile database operable to manage profile base objects corresponding to the sub-users;
a holdout compute engine operable to assign the profile base objects to a holdout group and to a non-holdout group;
a membership database operable to:
arrange and store the profile base objects of the sub-users according to holdout group and a non-holdout group;
publish assigned membership event base objects corresponding to each of the profile base objects in the holdout group and the profile base objects in the non-holdout group;
join, by sub-user, unassigned raw event base objects with the assigned membership event base objects and create an updated set of assigned event base objects for the holdout group and for the non-holdout group
a user interface module operable to request testing parameters comprising holdout percentage for a holdout group, and test duration; and
wherein the holdout compute engine is further operable to:
request, from the membership database, total counts of the assigned event base objects in the holdout group and total counts of the assigned event base objects in the non-holdout group;
request, from the membership database, conversion counts of the conversion events for the assigned event base objects in the holdout group and conversion counts for the conversion events for the assigned event base objects in the non-holdout group; and
compute at least one conversion rate based on the test duration, conversion counts, and total counts.

### Claim 12 (depends on 11)

The system of claim 11, further comprising a selector module operable for selecting electronic message sending strategies to override from the holdout group.

### Claim 13 (depends on 12)

The system of claim 12, wherein the messaging strategy types include one-time manual message sends and automatic sends based on trigger events.

### Claim 14 (depends on 12)

The system of claim 12, wherein the selecting is performed by a selector model operable to classify candidates to override from the holdout group.

### Claim 15 (depends on 11)

The system of claim 11, wherein the assigning is performed by a hashing algorithm.

### Claim 16 (depends on 15)

The system of claim 15, wherein the holdout compute engine is operable to assign in real-time within volatile memory structures without committing data to persistent storage mediums.

### Claim 17 (depends on 11)

The system of claim 11, wherein the holdout analytics server is further operable to compute a win probability.

### Claim 18 (depends on 12)

The system of claim 12, wherein the selecting comprises identifying messages automatically triggered based on a sub-user detected behavior.

### Claim 19 (depends on 11)

The system of claim 11, wherein the exclusion rules are based on excluding profile base objects from testing unless a first type of sub-user behavior is detected/confirmed.

### Claim 20

A non-transitory machine-readable medium, comprising executable instructions that, when executed by a processing system including a processor, perform operations for computing lift, the operations comprising:
selecting electronic message sending strategies to override from a holdout group;
retrieve, from a profile base object database valid profile base objects;
preprocessing the valid profile base objects based on a set of exclusion rules;
assigning a first set of profile base objects to the holdout group and a second set of profile base objects to the non-holdout group;
publishing asynchronously, in batches, assigned membership event base objects corresponding to the profile base objects in the holdout group and the profile base objects in the non-holdout group;
joining, by a profile ID, the assigned membership event base objects and unassigned raw event base objects received from a raw event pipeline, thereby creating an updated set of assigned event base objects in the holdout group and in the non-holdout group;
getting total counts of the assigned event base objects in the holdout group and total counts of the assigned event base objects in the non-holdout group;
getting conversion counts of conversion events of the assigned event base objects in the holdout group and conversion counts of the conversion events of the assigned event base objects in the non-holdout group; and
computing at least one conversion rate based on the test duration, conversion counts, and total counts.
