Systems and methods for providing event attribution in software applications

A method for event attribution during software experimentation, comprising:

• applying, by a server computer system, feature treatments to configurable applications executing end user systems associated with plurality of end users, the feature treatments specified by a customer system associated with a customer identifier, and the feature treatments configure one or more features of the configurable applications,
• receiving, by the server computer system, a plurality of event tracking messages associated with the end user systems, each event tracking message including at least the customer identifier, at least one end user key associated with a user system generating an event tracking message, and a timestamp;
• storing the event tracking messages with the received customer identifier, at least one end user key, and timestamp in a customer data store;
• receiving, from the customer system, an experiment rule defining how users are identified during an experiment, a selected feature treatment, and a time period of the experiment, wherein the experiment rule defining how users are identified during the experiment comprises one or more user key values, where the one or more user key values comprise values indicative of a user identifier, a class of user, a user account type, a user demographic, or a combination thereof,
• attributing, by the server computer system, a user to the experiment rule and the selected feature treatment for the time period of the experiment for which the experiment rule identifies the user, wherein the user is attributed to the experiment rule when, during the time period of the experiment, (1) an end user key value received in an event tracking message matches at least one of the one or more user key values defined by the experiment rule, and (2) the match occurs during the time period of the experiment;
• and
• attributing, by the server computer system, events received from event tracking messages generated by an end user system associated with the user when the user is attributed with the experiment rule and is exposed to the selected feature treatment during the time period of the experiment.

A non-transitory computer readable storage medium including instructions that, when executed by a processor, cause the processor to perform operations for event attribution during software experimentation, the operations comprising:

• applying, by a server computer system, feature treatments to configurable applications executing end user systems associated with plurality of end users, the feature treatments specified by a customer system associated with a customer identifier, and the feature treatments configure one or more features of the configurable applications;
• receiving, by the server computer system, a plurality of event tracking messages associated with the end user systems, each event tracking message including at least the customer identifier, at least one end user key associated with a user system generating an event tracking message, and a timestamp;
• storing the event tracking messages with the received customer identifier, at least one end user key, and timestamp in a customer data store;
• receiving, from the customer system, an experiment rule defining how users are identified during an experiment, a selected feature treatment, and a time period of the experiment;
• attributing, by the server computer system, a user to the experiment rule and the selected feature treatment for the time period of the experiment for which the experiment rule identifies the user;
• and
• attributing, by the server computer system, events received from event tracking messages generated by an end user system associated with the user when the user is attributed with the experiment rule and is exposed to the selected feature treatment during the time period of the experiment, further comprising:
• determining, for each of the received plurality of event tracking messages, when to attribute an event from an event tracking message with the selected feature treatment when there is a co-occurrence of:
• (1) an end user key in the event tracking message is identified by the experiment rule, (2) the event reported by the event tracking message occurs after a start time of the experiment and during the time period of the experiment, and (3) the configurable application at the end user system is exposed to the selected feature treatment during the time period of the experiment at which the user is attributed to the experiment rule, and
• attributing a set of events including the event from the received plurality of event tracking messages with the application of the selected feature treatment.

A non-transitory computer readable storage medium including instructions that, when executed by a processor, cause the processor to perform operations for event attribution during software experimentation, the operations comprising:

• applying, by a server computer system, feature treatments to configurable applications executing end user systems associated with plurality of end users, the feature treatments specified by a customer system associated with a customer identifier, and the feature treatments configure one or more features of the configurable applications;
• receiving, by the server computer system, a plurality of event tracking messages associated with the end user systems, each event tracking message including at least the customer identifier, at least one end user key associated with a user system generating an event tracking message, and a timestamp;
• storing the event tracking messages with the received customer identifier, at least one end user key, and timestamp in a customer data store;
• receiving, from the customer system, an experiment rule defining how users are identified during an experiment, a selected feature treatment, and a time period of the experiment, wherein the experiment rule defining how users are identified during the experiment comprises one or more user key values, where the one or more user key values comprise values indicative of a user identifier, a class of user, a user account type, a user demographic, or a combination thereof;
• attributing, by the server computer system, a user to the experiment rule and the selected feature treatment for the time period of the experiment for which the experiment rule identifies the user, wherein the user is attributed to the experiment rule when, during the time period of the experiment, (1) an end user key value received in an event tracking message matches at least one of the one or more user key values defined by the experiment rule, and (2) the match occurs during the time period of the experiment;
• and
• attributing, by the server computer system, events received from event tracking messages generated by an end user system associated with the user when the user is attributed with the experiment rule and is exposed to the selected feature treatment during the time period of the experiment.

A system for event attribution during software experimentation, comprising:

• a memory;
• and
• a processor coupled with the memory configured to:
• apply feature treatments to configurable applications executing end user systems associated with plurality of end users, the feature treatments specified by a customer system associated with a customer identifier, and the feature treatments configure one or more features of the configurable applications,
• receive a plurality of event tracking messages associated with the end user systems, each event tracking message including at least the customer identifier, at least one end user key associated with a user system generating an event tracking message, and a timestamp,
• store the event tracking messages with the received customer identifier, at least one end user key, and timestamp in a customer data store,
• receive, from the customer system, an experiment rule defining how users are identified during an experiment, a selected feature treatment, and a time period of the experiment,
• attribute a user to the experiment rule and the selected feature treatment for the time period of the experiment for which the experiment rule identifies the user, and
• attribute events received from event tracking messages generated by an end user system associated with the user when the user is attributed with the experiment rule and is exposed to the selected feature treatment during the time period of the experiment, further comprising the processor to:
• determine, for each of the received plurality of event tracking messages, when to attribute an event from an event tracking message with the selected feature treatment when there is a co-occurrence of:
• (1) an end user key in the event tracking message is identified by the experiment rule, (2) the event reported by the event tracking message occurs after a start time of the experiment and during the time period of the experiment, and (3) the configurable application at the end user system is exposed to the selected feature treatment during the time period of the experiment at which the user is attributed to the experiment rule, and
• attribute a set of events including the event from the received plurality of event tracking messages with the application of the selected feature treatment.

A system for event attribution during software experimentation, comprising:

• a memory;
• and
• a processor coupled with the memory configured to:
• apply feature treatments to configurable applications executing end user systems associated with plurality of end users, the feature treatments specified by a customer system associated with a customer identifier, and the feature treatments configure one or more features of the configurable applications;
• receive a plurality of event tracking messages associated with the end user systems, each event tracking message including at least the customer identifier, at least one end user key associated with a user system generating an event tracking message, and a timestamp;
• store the event tracking messages with the received customer identifier, at least one end user key, and timestamp in a customer data store;
• receive, from the customer system, an experiment rule defining how users are identified during an experiment, a selected feature treatment, and a time period of the experiment, wherein the experiment rule defining how users are identified during the experiment comprises one or more user key values, where the one or more user key values comprise values indicative of a user identifier, a class of user, a user account type, a user demographic, or a combination thereof;
• attribute a user to the experiment rule and the selected feature treatment for the time period of the experiment for which the experiment rule identifies the user, wherein the user is attributed to the experiment rule when, during the time period of the experiment, (1) an end user key value received in an event tracking message matches at least one of the one or more user key values defined by the experiment rule, and (2) the match occurs during the time period of the experiment;
• and
• attribute events received from event tracking messages generated by an end user system associated with the user when the user is attributed with the experiment rule and is exposed to the selected feature treatment during the time period of the experiment.