# US 11,909,829 B1 · Online testing efficiency through early termination > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,909,829 B1 - Document ID: US11909829B1 - Status: Granted - Kind: B1 - Application number: 18/336,802 - Filed: 2023-06-16 - Granted: 2024-02-20 - Earliest claimed: 2023-06-16 - Assignees: Amazon Technologies, Inc. - Applicants: Amazon Technologies, Inc. - Inventors: Yu Liu; Runzhe Wan; James McQueen; Doug Hains; Rui Song; Roger Henry Castillo - Web page: https://patentgrep.com/patent/US11909829B1 ## CPC - H04L 67/143: Termination or inactivation of sessions, e.g. event-controlled end of session - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract The present disclosure generally relates to systems and methods for generating termination notification of an experiment to be presented through a user interface. In some implementation examples, a termination notification system generates a probability value that is valid as of a time period of an experiment at a predetermined point in time based on at least data sample obtained during the time period, predicted sample sizes of each time period of the experiment, or probability values generated prior to the time period. Responsive to determining that the probability value that is valid as of the time period satisfies a threshold value, the termination notification system causes the user interface to present the termination notification that, when selected, causes the experiment to terminate prior to completion of a predicted duration of the experiment. ## Claims ### Claim 1 A computer-implemented method comprising: obtaining first data during a first time period of execution of an experiment; determining that a predicted duration of the experiment includes the first time period and subsequent time periods that end in time after the first time period; determining a predicted sample size of each of the first time period and the subsequent time periods; for a second time period of the subsequent time periods that ends in time after the first time period: obtaining subsequent data during the second time period; generating a second probability value of the experiment that is valid as of the second time period based on at least one of (i) the subsequent data obtained during the second time period, (ii) the predicted sample size of each of the first time period and the subsequent time periods, or (iii) individual second probability values valid as of each of the subsequent time periods that precede in time the second time period; determining that the second probability value of the experiment that is valid as of the second time period satisfies a threshold value; and in response to determining that the second probability value of the experiment that is valid as of the second time period satisfies the threshold value, causing a user interface to display a termination notification, wherein the termination notification, when selected, causes the experiment to terminate prior to completion of the predicted duration. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising generating a first probability value of the experiment that is valid as of the first time period based on at least one of (i) the first data or (ii) the predicted sample size of each of the first time period and the subsequent time periods. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein the first probability value of the experiment that is valid as of the first time period is a p-value. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, wherein the p-value is associated with an alpha spending function. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein an information fraction of the alpha spending function is the predicted sample size of the first time period divided by a predicted sample size of the experiment. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, further comprising determining, based on the first data, a predicted statistical result of the experiment, wherein the predicted statistical result of the experiment comprises at least one of a predicted sample size of the experiment, a predicted mean of the predicted sample size of the experiment, or a predicted variance of the predicted sample size of the experiment. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein the predicted duration of the experiment is determined based on the predicted statistical result. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the second probability value of the experiment that is valid as of the second time period is generated at an end of the second time period. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the threshold value is adjustable through the user interface during the predicted duration of the experiment. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein each of the first time period and the subsequent time periods is non-overlapping in time with each other. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, further comprising in response to receiving a selection of a user interface element of the user interface, causing the user interface to display the individual second probability values valid as of each of the subsequent time periods that precede in time the second time period. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein the first time period is a subset in time of each of the subsequent time periods. ### Claim 13 A system for presenting a termination notification of an experiment comparing a first version of a content page with a second version of the content page through a user interface, the system comprising: one or more processors; and a memory that stores computer-executable instructions, wherein the computer-executable instructions, when executed, cause the one or more processors to: obtain first data during a first time period of execution of the experiment; determine, based on the first data, a predicted statistical result of the experiment; determine, based on the predicted statistical result, that a predicted duration of the experiment includes the first time period and subsequent time periods that end in time after the first time period; determine a predicted sample size of each of the first time period and the subsequent time periods; generate a first probability value of the experiment that is valid as of the first time period based on at least one of (i) the first data or (ii) the predicted sample size of each of the first time period and the subsequent time periods; for a second time period of the subsequent time periods that ends in time after the first time period: obtain subsequent data during the second time period; generate a second probability value of the experiment that is valid as of the second time period based on at least one of (i) the subsequent data obtained during the second time period, (ii) the predicted sample size of each of the first time period and the subsequent time periods, (iii) the first probability value, or (iv) individual second probability values valid as of each of the subsequent time periods that precede in time the second time period; determine that the second probability value of the experiment that is valid as of the second time period satisfies a threshold value; and in response to determining that the second probability value of the experiment that is valid as of the second time period satisfies the threshold value, cause the user interface to display the termination notification, wherein the termination notification, when selected, causes the experiment to terminate prior to completion of the predicted duration. ### Claim 14 (depends on 13) The system of claim 13, wherein the first probability value of the experiment that is valid as of the first time period is generated at an end of the first time period. ### Claim 15 (depends on 13) The system of claim 13, wherein the predicted statistical result of the experiment comprises at least one of a predicted sample size of the experiment, a predicted mean of the predicted sample size of the experiment, or a predicted variance of the predicted sample size of the experiment. ### Claim 16 (depends on 13) The system of claim 13, wherein the first probability value of the experiment that is valid as of the first time period is a p-value. ### Claim 17 (depends on 13) The system of claim 13, wherein the first time period is a first week of the experiment. ### Claim 18 One or more non-transitory computer-readable media comprising computer-executable instructions for presenting a termination notification of an experiment, wherein the computer-executable instructions, when executed by a computer system, cause the computer system to perform operations comprising: obtaining first data during a first time period of execution of the experiment; determining that a predicted duration of the experiment includes the first time period and a second time period; determining a predicted sample size of each of the first time period and the second time period; for the second time period: obtaining subsequent data during the second time period; generating a second probability value of the experiment that is valid as of the second time period; determining that the second probability value of the experiment that is valid as of the second time period satisfies a threshold value; and in response to determining that the second probability value of the experiment that is valid as of the second time period satisfies the threshold value, cause a user interface to display the termination notification, wherein the termination notification, when selected, causes the experiment to terminate prior to completion of the predicted duration. ### Claim 19 (depends on 18) The one or more non-transitory computer-readable media of claim 18, further comprising generating a first probability value of the experiment that is valid as of the first time period based on at least one of (i) the first data or (ii) the predicted sample size of each of the first time period and the second time period. ### Claim 20 (depends on 19) The one or more non-transitory computer-readable media of claim 19, wherein the first time period is one of partially overlapping in time with the second time period, a subset of the second time period, non-overlapping and contiguous with the second time period, or non-overlapping and non-contiguous with the second time period.