Methods and systems for generating values of overall evaluation criterion

A method of generating values of an overall evaluation criterion (OEC) to be used during a control/treatment testing of a computer-implemented service for selecting a current version of the computer-implemented service amongst a control and a treatment version of the computer-implemented service, the method being executable by a server, the method comprising:

• for each one of the control and treatment versions:
• receiving, by the server, a respective dataset indicative of user interactions of users with the respective one of the control and treatment versions;
• determining, by the server, a respective first set of values based on the respective dataset, values of the first sets being values of a first per-user metric;
• and
• determining, by the server, a respective second set of values based on the respective dataset, values of the second sets being values of a second per-user metric;
• selecting, by the server, a linearization value for a linearized per-user metric, the linearized per-user metric being a linear combination of the first and second per-user metrics, the OEC being based on the linearized per-user metric, the linearization value being between a control-ratio and a treatment-ratio,
• the control-ratio being a ratio of (i) an average value of the values of the first set of the control version over (ii) an average value of the values of the second set of the control version,
• the treatment-ratio being a ratio of (i) an average value of the values of the first set of the treatment version over (ii) an average value of the values of the second set of the treatment version;
• for each one of the control and treatment versions:
• determining, by the server, a respective linearized set of values, values in a given linearized set being differences between (i) the values of the respective first set from (ii) the values of the respective second set that are weighted by the linearization value, the values of the linearized sets being values of the linearized per-user metric;
• and
• generating, by the server, a respective value of the OEC based on the respective linearized sets, whereby a difference between the values of the OEC is indicative of (i) a magnitude of the change in the user interactions between the control version and the treatment version and (ii) a direction of the change in the user interactions between the control version and the treatment version, the direction being one of a positive direction and a negative direction;
• selecting, by the server, one of the control version and the treatment version as the current version of the computer-implemented service based on the direction of change in the user interactions between the control and the treatment versions;
• and
• providing, by the server, the current version of the computer-implemented service to the users of the computer-implemented service.