Automatic experience research with a user personalization option method and apparatus

A non-transitory computer-readable storage medium tangibly encoded with computer-executable instructions that when executed by a processor associated with a computing device perform a method comprising:

• receiving an evaluation request in connection with a user experience experiment designed to adopt a global-best user experience variant, from a number of user experience variants, of a software application's user interface, the user experience experiment involving each of a number of user groups being presented with a corresponding experience variant of the software application's user interface;
• forming a training user group and an evaluation user group using a pool of users from the number of user groups;
• obtaining, for each user in the user pool, user data comprising, for each user, an experience variant designation and a corresponding user response metric indicating the user's response, capturing via the software application, to a respective user experience variant of the software application's user interface presented, by the software application, to the user, at a computing device of the user, during the user experience experiment, the user data further comprising a number of user attributes;
• using the experience variant designation and the corresponding user response metric obtained for each user in the evaluation user group to determine a number of aggregate user responses corresponding to the number of user experience variants;
• using the number of aggregate user responses to identify one of the number of user experience variants with a corresponding aggregate user response greater than each other of the number of aggregate user response as the global-best user experience variant for evaluation;
• generating training data using the user data corresponding to each user assigned to the training user group;
• training a user response prediction model using a machine learning algorithm and the training data;
• using the trained user response prediction model to determine, for each user in the evaluation user group, a variant preference prediction identifying one of the number of user experience variants and a corresponding user response prediction;
• determining an aggregate user response prediction using the user response prediction determined for each user in the evaluation user group;
• automatically evaluating an ability of the global-best user experience variant at optimizing user response relative to the trained user response prediction model's ability using the global-best user experience variant's corresponding aggregate user response and the aggregate user response prediction;
• identifying, from the number of user experience variants, a user experience variant that, based on the evaluation, optimizes the software application's user response;
• and
• presenting, by the software application, the identified user experience variant is used by the software application as its user interface at a number of user computing devices.

A computing device comprising:

• a processor;
• a non-transitory storage medium for tangibly storing thereon program logic for execution by the processor, the program logic comprising:
• receiving logic executed by the processor for receiving an evaluation request in connection with a user experience experiment designed to adopt a global-best user experience variant, from a number of user experience variants, of a software application's user interface, the user experience experiment involving each of a number of user groups being presented with a corresponding user experience variant of the software application's user interface;
• forming logic executed by the processor for forming a training user group and an evaluation user group using a pool of users from the number of user groups;
• obtaining logic executed by the processor for obtaining, for each user in the user pool, user data comprising, for each user, an experience variant designation and a corresponding user response metric indicating the user's response, captured via the software application, to a respective user experience variant of the software application's user interface presented, by the software application, to the user, at a computing device of the user, during the user experience experiment, the user data further comprising a number of user attributes;
• using logic executed by the processor for using the experience variant designation and the corresponding user response metric obtained for each user in the evaluation user group to determine a number of aggregate user responses corresponding to the number of user experience variants;
• using logic executed by the processor for using the number of aggregate user responses to identify one of the number of user experience variants with a corresponding aggregate user response greater than each other of the number of aggregate user response as the global-best user experience variant for evaluation;
• generating logic executed by the processor for generating training data using the user data corresponding to each user assigned to the training user group;
• training logic executed by the processor for training a user response prediction model using a machine learning algorithm and the training data;
• using logic executed by the processor for using the trained user response prediction model to determine, for each user in the evaluation user group, a variant preference prediction identifying one of the number of user experience variants and a corresponding user response prediction;
• determining logic executed by the processor for determining an aggregate user response prediction using the user response prediction determined for each user in the evaluation user group;
• evaluating logic executed by the processor for automatically evaluating an ability of the global-best user experience variant at optimizing user response relative to the trained user response prediction model's ability using the global-best user experience variant's corresponding aggregate user response and the aggregate user response prediction;
• identifying logic executed by the processor for identifying, from the number of user experience variants, a user experience variant that, based on the evaluation, optimized the software application's user response, the identified user experience variant being presented, by the software application, as its user interface at a number of user computing devices.