Computer-implemented Method for Global Holdout Testing and Related Systems

A system for computing lift for global holdout testing, wherein the global holdout testing measures conversions for multiple types of electronic messages and sending strategies, the system comprising:

• a raw event database operable to manage raw event base objects arising from sub-user behaviors;
• a profile database operable to manage profile base objects corresponding to the sub-users;
• a holdout compute engine operable to assign the profile base objects to a holdout group and to a non-holdout group;
• a membership database operable to:
• arrange and store the profile base objects of the sub-users according to holdout group and a non-holdout group;
• publish assigned membership event base objects corresponding to each of the profile base objects in the holdout group and the profile base objects in the non-holdout group;
• join, by sub-user, unassigned raw event base objects with the assigned membership event base objects and create an updated set of assigned event base objects for the holdout group and for the non-holdout group
• a user interface module operable to request testing parameters comprising holdout percentage for a holdout group, and test duration;
• and
• wherein the holdout compute engine is further operable to:
• request, from the membership database, total counts of the assigned event base objects in the holdout group and total counts of the assigned event base objects in the non-holdout group;
• request, from the membership database, conversion counts of the conversion events for the assigned event base objects in the holdout group and conversion counts for the conversion events for the assigned event base objects in the non-holdout group;
• and
• compute at least one conversion rate based on the test duration, conversion counts, and total counts.

A non-transitory machine-readable medium, comprising executable instructions that, when executed by a processing system including a processor, perform operations for computing lift, the operations comprising:

• selecting electronic message sending strategies to override from a holdout group;
• retrieve, from a profile base object database valid profile base objects;
• preprocessing the valid profile base objects based on a set of exclusion rules;
• assigning a first set of profile base objects to the holdout group and a second set of profile base objects to the non-holdout group;
• publishing asynchronously, in batches, assigned membership event base objects corresponding to the profile base objects in the holdout group and the profile base objects in the non-holdout group;
• joining, by a profile ID, the assigned membership event base objects and unassigned raw event base objects received from a raw event pipeline, thereby creating an updated set of assigned event base objects in the holdout group and in the non-holdout group;
• getting total counts of the assigned event base objects in the holdout group and total counts of the assigned event base objects in the non-holdout group;
• getting conversion counts of conversion events of the assigned event base objects in the holdout group and conversion counts of the conversion events of the assigned event base objects in the non-holdout group;
• and
• computing at least one conversion rate based on the test duration, conversion counts, and total counts.