Model-based matching for removing selection bias in quasi-experimental testing of mobile applications

An apparatus, comprising:

• one or more processors;
• and
• memory storing instructions that, when executed by the one or more processors, cause the apparatus to:
• obtain, during a partial rollout of a new version of a mobile application, a first set of data associated with a set of adopters of the new version and a second set of data associated with a set of non-adopters of the new version, wherein the partial rollout of the new version of the mobile application releases the new version to a random set of users;
• apply a statistical model to the first and second sets of data to select a set of potential adopters of the new version among the set of non-adopters, wherein the set of potential adopters represents users who are inclined to adopt the new version but are currently non-adopters,
• wherein the statistical model includes a matching model, and wherein applying the statistical model to select the set of potential adopters includes:
• comparing the first set of data to the second set of data using the matching model to generate a set of matches among the set of adopters and a subset of the set of non-adopters;
• and
• selecting the subset of the set of non-adopters as the set of potential adopters;
• and
• reduce a bias in a quasi-experimental design associated with the mobile application by using the first set of data associated with the set of adopters and a third set of data associated with the set of potential adopters to estimate an average treatment effect (ATE) between the new version and an older version of the mobile application.

A system, comprising:

• a validation apparatus comprising a non-transitory computer-readable medium comprising instructions that, when executed, cause the system to:
• obtain, during a partial rollout of a new version of a mobile application, a first set of data associated with a set of adopters of the new version and a second set of data associated with a set of non-adopters of the new version, wherein the partial rollout of the new version of the mobile application releases the new version to a random set of users;
• apply a statistical model to the first and second sets of data to select a set of potential adopters of the new version among the set of non-adopters, wherein the set of potential adopters represents users who are inclined to adopt the new version but are currently non-adopters,
• wherein the statistical model includes a matching model, and wherein applying the statistical model to select the set of potential adopters includes:
• comparing the first set of data to the second set of data using the matching model to generate a set of matches among the set of adopters and a subset of the set of non-adopters;
• and
• selecting the subset of the set of non-adopters as the set of potential adopters;
• and
• reduce a bias in a quasi-experimental design associated with the mobile application by using the first set of data associated with the set of adopters and a third set of data associated with the set of potential adopters to estimate an average treatment effect (ATE) between the new version and an older version of the mobile application.