# US 10,372,599 B2 · Model-based matching for removing selection bias in quasi-experimental testing of mobile applications > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,372,599 B2 - Document ID: US10372599B2 - Status: Granted - Kind: B2 - Application number: 15/140,239 - Filed: 2016-04-27 - Published: 2017-11-02 - Granted: 2019-08-06 - Earliest claimed: 2016-04-27 - Assignees: Microsoft Technology Licensing, LLC - Applicants: LinkedIn Corporation - Inventors: Ya Xu; Nanyu Chen; Bryan G. Ng; Weitao Duan - Web page: https://patentgrep.com/patent/US10372599B2 ## CPC - G06F 11/3692: {for test results analysis} - G06F 9/50: Allocation of resources, e.g. of the central processing unit [CPU] - G06F 9/54: Interprogram communication - G06F 11/3672: {Test management} - G06F 11/368: {for test version control, e.g. updating test cases to a new software version} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract The disclosed embodiments provide a system for evaluating a performance of a mobile application. During operation, the system obtains a first set of data associated with adopters of a new version of a mobile application in a partial rollout of the new version and a second set of data associated with non-adopters of the new version in the partial rollout. Next, the system applies a statistical model to the first and second sets of data to select a subset of the non-adopters as potential adopters of the new version. The system then reduces a bias in a quasi-experimental design associated with the mobile application by using the first set of data and a third set of data associated with the potential adopters to estimate an average treatment effect (ATE) between the new version and an older version of the mobile application. ## Claims ### Claim 1 A method, comprising: obtaining, during a partial rollout of a new version of a mobile application, a first set of data associated with a set of adopters of the new version and a second set of data associated with a set of non-adopters of the new version, wherein the partial rollout of the new version of the mobile application releases the new version to a random set of users; applying, by a computer system, a statistical model to the first and second sets of data to select a set of potential adopters of the new version among the set of non-adopters, wherein the set of potential adopters represents users who are inclined to adopt the new version but are currently non-adopters, wherein the statistical model includes a matching model, and wherein applying the statistical model to select the set of potential adopters includes: comparing the first set of data to the second set of data using the matching model to generate a set of matches among the set of adopters and a subset of the set of non-adopters; and selecting the subset of the set of non-adopters as the set of potential adopters; and reducing a bias in a quasi-experimental design associated with the mobile application by using the first set of data associated with the set of adopters and a third set of data associated with the set of potential adopters to estimate an average treatment effect (ATE) between the new version and an older version of the mobile application. ### Claim 2 (depends on 1) The method of claim 1, wherein applying the matching model to the first and second sets of data to generate the set of matches comprises: using a subset of covariates in the first and second sets of data to perform exact matching of a subset of the set of adopters to the subset of the set of non-adopters. ### Claim 3 (depends on 2) The method of claim 2, wherein applying the matching model to the first and second sets of data to match the set of adopters to the subset of the set of non-adopters further comprises: selecting a number of covariates used in the matching model based on a number of adopters of the new version. ### Claim 4 (depends on 2) The method of claim 2, wherein the subset of covariates comprises a bucketized variable. ### Claim 5 (depends on 1) The method of claim 1, wherein applying the matching model to the first and second sets of data to match the set of adopters to the subset of the set of non-adopters comprises: for each match in the set of matches, assigning a weight to users in the match based on a number of the set of adopters and a number of the set of nonadopters in the match. ### Claim 6 (depends on 1) The method of claim 1, wherein using the first and third sets of data to estimate the ATE between the new version and the older version of the application comprises: producing one or more weighted linear models from the set of matches and a full set of covariates in the first and third sets of data; and using the one or more weighted linear models to estimate the ATE. ### Claim 7 (depends on 6) The method of claim 6, wherein the one or more weighted linear models comprise an endogenous switching model. ### Claim 8 (depends on 1) The method of claim 1, further comprising: using the estimated ATE to evaluate a novelty effect associated with adoption of the new version. ### Claim 9 (depends on 8) The method of claim 8, wherein using the estimated ATE to evaluate the novelty effect associated with adoption of the new version comprises: estimating the ATE from matched cohorts of the set of adopters and the set of non-adopters over an evaluation period of the quasi-experimental design. ### Claim 10 (depends on 1) The method of claim 1, further comprising: using historic data collected from substantially identical versions of the mobile application to validate the statistical model. ### Claim 11 An apparatus, comprising: one or more processors; and memory storing instructions that, when executed by the one or more processors, cause the apparatus to: obtain, during a partial rollout of a new version of a mobile application, a first set of data associated with a set of adopters of the new version and a second set of data associated with a set of non-adopters of the new version, wherein the partial rollout of the new version of the mobile application releases the new version to a random set of users; apply a statistical model to the first and second sets of data to select a set of potential adopters of the new version among the set of non-adopters, wherein the set of potential adopters represents users who are inclined to adopt the new version but are currently non-adopters, wherein the statistical model includes a matching model, and wherein applying the statistical model to select the set of potential adopters includes: comparing the first set of data to the second set of data using the matching model to generate a set of matches among the set of adopters and a subset of the set of non-adopters; and selecting the subset of the set of non-adopters as the set of potential adopters; and reduce a bias in a quasi-experimental design associated with the mobile application by using the first set of data associated with the set of adopters and a third set of data associated with the set of potential adopters to estimate an average treatment effect (ATE) between the new version and an older version of the mobile application. ### Claim 12 (depends on 11) The apparatus of claim 11, wherein applying the matching model to the first and second sets of data to generate the set of matches comprises: using a subset of covariates in the first and second sets of data to perform exact matching of a subset of the set of adopters to the subset of the set of non-adopters. ### Claim 13 (depends on 12) The apparatus of claim 12, wherein applying the matching model to the first and second sets of data to match the set of adopters to the subset of the set of non-adopters further comprises: selecting a number of covariates used in the matching model based on a number of adopters of the new version. ### Claim 14 (depends on 11) The apparatus of claim 11, wherein applying the matching model to the first and second sets of data to match the set of adopters to the subset of the set of non-adopters comprises: for each match in the set of matches, assigning a weight to users in the match based on a number of the set of adopters and a number of the set of nonadopters in the match. ### Claim 15 (depends on 11) The apparatus of claim 11, wherein using the first and third sets of data to estimate the ATE between the new version and the older version of the application comprises: producing one or more weighted linear models from the set of matches and a full set of covariates in the first and third sets of data; and using the one or more weighted linear models to estimate the ATE. ### Claim 16 (depends on 11) The apparatus of claim 11, wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to: use the estimated ATE to evaluate a novelty effect associated with adoption of the new version. ### Claim 17 A system, comprising: a validation apparatus comprising a non-transitory computer-readable medium comprising instructions that, when executed, cause the system to: obtain, during a partial rollout of a new version of a mobile application, a first set of data associated with a set of adopters of the new version and a second set of data associated with a set of non-adopters of the new version, wherein the partial rollout of the new version of the mobile application releases the new version to a random set of users; apply a statistical model to the first and second sets of data to select a set of potential adopters of the new version among the set of non-adopters, wherein the set of potential adopters represents users who are inclined to adopt the new version but are currently non-adopters, wherein the statistical model includes a matching model, and wherein applying the statistical model to select the set of potential adopters includes: comparing the first set of data to the second set of data using the matching model to generate a set of matches among the set of adopters and a subset of the set of non-adopters; and selecting the subset of the set of non-adopters as the set of potential adopters; and reduce a bias in a quasi-experimental design associated with the mobile application by using the first set of data associated with the set of adopters and a third set of data associated with the set of potential adopters to estimate an average treatment effect (ATE) between the new version and an older version of the mobile application. ### Claim 18 (depends on 17) The system of claim 17, wherein applying the matching model to the first and second sets of data to generate the set of matches comprises: using a subset of covariates in the first and second sets of data to perform exact matching of a subset of the set of adopters to the subset of the set of non-adopters.