US 12,725,177 B2Granted

Systems, methods, and media for comparing variations of electronic content in relation to a measured metric and/or generating a causal model to calculate the effect of an input variable on a desired outcome

AssigneeFmr LLC

InventorsRaj Devesh, Ankit Ashok Baraskar, Manish Worlikar

Life of this patent

Estimating the term…

Loading patent…

Abstract

In an embodiment, a feature that is different between variations of web-based electronic content can be determined to be a true and significant cause of a difference of a measured metric when three conditions are met. The three conditions may be satisfied when a probability value for a false positive is less than a false positive threshold value, a statistical power is greater than a statistical power threshold value, and a difference between mean values determined for distributions is greater than a mean threshold value. In a further embodiment, and when there is no control group, a causal model can be generated for different interventions or values related to a desired outcome. The causal model can be executed to determine which intervention is most impactful on the desired outcome and/or to determine an overall impact of a categorical variable on the desired outcome.

Claims16

Dependent claims

A computer implemented method for performing A/B testing for a first variation of web-based electronic content and a second variation of the web-based electronic content, the method comprising:

receiving, from a client device operated by a user, A/B testing input information for the first variation of the web-based electronic content and the second variation of the web-based electronic content, wherein the first variation of the web-based electronic content is utilized by a first group of a plurality of first users for the A/B testing and the second variation of the web-based electronic content is utilized by a second group of a plurality of second users for the A/B testing;
determining, based on the A/B testing input information, a type of a variable corresponding to a feature that is different in the first variation of the web-based electronic content and the second variation of the web-based electronic content;
determining a distribution of a performance metric among the plurality of first users and among the plurality of second users, wherein the performance metric is related to the variable corresponding to the feature;
identifying a statistical test based on the determined distribution among the plurality of first users and the plurality of second users;
computing, for the statistical test, a first probability value for a false positive in performance of the A/B testing;
computing, for the statistical test, a second probability value for a false negative in performance of the A/B testing;
subtracting, for the statistical test, the second probability value from 1 to produce a statistical power for the A/B testing;
computing a first mean value for the distribution of the performance metric among the plurality of first users and a second mean value for the distribution of the performance metric among the plurality of second users;
defining a difference in the performance metric, across the first group utilizing the first variation and the second group utilizing the second variation, as significant when (1) the first probability is less than a first threshold value, (2) the difference between the first mean value and the second mean value is greater than a second threshold value, and (3) the statistical power is greater than a third threshold value;
detecting, by a software module executed by a processor of a computing device, that (1) the first probability is less than the first threshold value, (2) the difference between the first mean value and the second mean value is greater than the second threshold value, and (3) the statistical power is greater than a third threshold value;
and
automatically updating, by the software module and in response to the detecting, a web page to include the web-based electronic content with the feature.

9 more elements

A system for performing A/B testing for a first variation of electronic web-based content and a second variation of the electronic web-based content, the system comprising:

a memory;
a processor coupled to the memory, the processor executing a software module configured to:
receive, from a client device operated by a user, A/B testing input information for the first variation of the electronic web-based content and the second variation of the electronic web-based content, wherein the first variation of the electronic web-based content is utilized by a first group of a plurality of first users for the A/B testing and the second variation of the electronic web-based content is utilized by a second group of a plurality of second users for the A/B testing;
determine, based on the A/B testing input information, a type of a variable corresponding to a feature that is different in the first variation of the electronic web-based content and the second variation of the electronic web-based content;
determine a distribution of a performance metric among the plurality of first users and among the plurality of second users, wherein the performance metric is related to the variable corresponding to the feature;
identify a statistical test based on the determined distribution among the plurality of first users and the plurality of second users;
compute, for the statistical test, a first probability value for a false positive in performance of the A/B testing;
compute, for the statistical test, a second probability value for a false negative in performance of the A/B testing;
subtract, for the statistical test, the second probability value from 1 to produce a statistical power for the A/B testing;
compute a first mean value for the distribution of the performance metric among the plurality of first users and a second mean value for the distribution of the performance metric among the plurality of second users;
define a difference in the performance metric across the first group utilizing the first variation and the second group utilizing the second variation as significant when (1) the first probability is less than a first threshold value, (2) the difference between the first mean value and the second mean value is greater than a second threshold value, and (3) the statistical power is greater than a third threshold value;
and
detect that (1) the first probability is less than the first threshold value, (2) the difference between the first mean value and the second mean value is greater than the second threshold value, and (3) the statistical power is greater than a third threshold value;
and
automatically update, in response to the detecting, a web page to include the electronic web-based content with the feature.

12 more elements

Dependencies are inferred from the claim text.