Assigning control and variants to dynamically defined time periods in application state experiments

A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:

• receiving data for an application state experiment for a client application of an online system, wherein the data for the application state experiment comprises a control application state for the client application, a test application state of the client application, a testing time period, and an interaction data criteria indicating a minimum number of interaction data points to be collected for each of the control application state and the test application state;
• conducting the application state experiment by, for each of a plurality of time periods within the testing time period:
• assigning one of:
• the control application state or the test application state to the time period;
• transmitting instructions to a plurality of client devices to cause the client application on the plurality of client devices to present a user interface in accordance with the assigned application state;
• collecting a set of interaction data points describing user interactions with the user interface of the assigned application state;
• ending the time period when the collected set of interaction data points meets the interaction data criteria;
• and
• collecting another set of interaction data points during a transition time period following the time period;
• computing a success metric for each of the control application state and the test application state based on the sets of interaction data points for the application states collected during the time periods, wherein the success metric is not computed based on the set of interaction data points collected during transition time periods
• transmitting instructions to a client device of the plurality of client devices to cause the client application on the client device to present a user interface in accordance with the control application state or the test application state based on the success metric.

A system comprising a processor and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the processor to perform operations comprising:

• receiving data for an application state experiment for a client application of an online system, wherein the data for the application state experiment comprises a control application state for the client application, a test application state of the client application, a testing time period, and an interaction data criteria indicating a minimum number of interaction data points to be collected for each of the control application state and the test application state;
• conducting the application state experiment by, for each of a plurality of time periods within the testing time period:
• assigning one of:
• the control application state or the test application state to the time period;
• transmitting instructions to a plurality of client devices to cause the client application on the plurality of client devices to present a user interface in accordance with the assigned application state;
• collecting a set of interaction data points describing user interactions with the user interface of the assigned application state;
• ending the time period when the collected set of interaction data points meets the interaction data criteria;
• and
• collecting another set of interaction data points during a transition time period following the time period;
• computing a success metric for each of the control application state and the test application state based on the sets of interaction data points for the application states collected during the time periods, wherein the success metric is not computed based on the set of interaction data points collected during transition time periods
• transmitting instructions to a client device of the plurality of client devices to cause the client application on the client device to present a user interface in accordance with the control application state or the test application state based on the success metric.