# US 10,313,262 B1 · System for management of content changes and detection of novelty effects > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,313,262 B1 - Document ID: US10313262B1 - Status: Granted - Kind: B1 - Application number: 15/385,178 - Filed: 2016-12-20 - Granted: 2019-06-04 - Earliest claimed: 2016-12-20 - Assignees: Amazon Technologies, Inc. - Applicants: Amazon Technologies, Inc. - Inventors: Alexandra Juliet Brasch; Brent Russell Smith; Douglas R. Hains; Corry Lee-Boehm - Web page: https://patentgrep.com/patent/US10313262B1 ## CPC - H04L 67/535: {Tracking the activity of the user (network monitoring arrangements H04L43/00; recording of computer activity G06F11/34)} - H04L 47/70: Admission control; Resource allocation - H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) - H04L 67/565: Conversion or adaptation of application format or content (adding application control or application functional data H04L67/561) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract Systems are provided to analyze user behavior with respect to different versions of content, and to identify the behavioral effects that can be attributed to the differences between the versions of content. Systems can determine whether an observed difference in user interactions with different versions of content is caused by the specific changes that have been made to content (e.g., an actual improvement or other substantive change), or whether the observed difference in user interactions is caused by the mere fact that the versions of content are different (e.g., due to the novelty of seeing a different version of content for the first time, regardless of whether the difference is a substantive improvement). In some cases, the observed effect may be caused both by the novelty of the content being changed and by the substance of the changes. ## Claims ### Claim 1 A computer-implemented method comprising: as performed by a computing system configured to execute specific instructions, determining an initial effect that a difference between a first version of a network resource and a second version of the network resource has on a metric during an initial period of time, wherein the metric relates to actions performed after accessing the network resource, and wherein the difference comprises at least one of: a portion of the first version is generated using a different algorithm than a corresponding portion of the second version, the portion of first version is presented in a different manner than the corresponding portion of the second version, or interaction with the portion of the first version causes a different action than interaction with the corresponding portion of the second version; determining a number of exposures to the network resource occurring during a subsequent period of time following the initial period of time; determining an expected effect that the difference between the first version and the second version is likely to have on the metric during the subsequent period of time, wherein the expected effect is determined based at least in part on the initial effect and the number of exposures; determining an observed effect that the difference between the first version and the second version has on the metric during the subsequent period of time, wherein the observed effect is determined based at least in part on data regarding actions performed during the subsequent period of time, and wherein an action performed by an individual user of the network resource during the subsequent period of time is performed after a user device of the individual user receives either the first version or the second version; determining to modify the network resource based at least in part on a degree the observed effect differs from the expected effect; and modifying the network resource. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the network resource comprises at least one of an audio presentation, a visual presentation, or a tactile presentation. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising receiving data regarding actions performed after exposure to the network resource during the initial period of time, wherein an action performed by a first user during the initial period of time is performed after a first user device of the first user receives the first version, and wherein an action performed by a second user during the initial period of time is performed after a second user device of the second user receives the second version. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the metric relates to at least one of: selections of a portion of the first version or the second version; transactions initiated using the network resource; or actions performed after receiving the network resource. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, further comprising: determining a second expected effect that the difference between the first version and the second version is likely to have on a second metric during the subsequent period of time; and determining a second observed effect that the difference between the first version and the second version has on the second metric during the subsequent period of time, wherein determining to modify the network resource is further based at least in part on a degree to which the second observed effect differs from the second expected effect. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, further comprising determining a rate at which a difference between the observed effect and the expected effect changes over the subsequent period of time. ### Claim 7 A non-transitory computer storage medium storing executable code, wherein the executable code configures a computing system to perform a process comprising: determining an initial effect that a difference between a first version of a network resource and a second version of the network resource has on a metric during an initial period of time, wherein the metric relates to actions performed after accessing the network resource, and wherein the difference comprises at least one of: a portion of the first version is generated using a different algorithm than a corresponding portion of the second version, the portion of first version is presented in a different manner than the corresponding portion of the second version, or interaction with the portion of the first version causes a different action than interaction with the corresponding portion of the second version; determining a number of exposures to the network resource occurring during a subsequent period of time following the initial period of time; determining an expected effect that the difference between the first version and the second version is likely to have on the metric during the subsequent period of time, wherein the expected effect is determined based at least in part on the initial effect and the number of exposures; determining an observed effect that the difference between the first version and the second version has on the metric during the subsequent period of time, wherein the observed effect is determined based at least in part on data regarding actions performed during the subsequent period of time, and wherein an action performed by an individual user of the network resource during the subsequent period of time is performed after a user device of the individual user receives either the first version or the second version; determining to modify the network resource based at least in part on a degree the observed effect differs from the expected effect; and modifying the network resource. ### Claim 8 (depends on 7) The non-transitory computer storage medium of claim 7, wherein the network resource comprises at least one of an audio presentation, a visual presentation, or a tactile presentation. ### Claim 9 (depends on 7) The non-transitory computer storage medium of claim 7, wherein the metric relates to at least one of: selections of a portion of the first version or the second version; transactions initiated using the network resource; or actions performed after receiving the network resource. ### Claim 10 (depends on 7) The non-transitory computer storage medium of claim 7, wherein the process further comprises: determining a second expected effect that the difference between the first version and the second version is likely to have on a second metric during the subsequent period of time; and determining a second observed effect that the difference between the first version and the second version has on the second metric during the subsequent period of time, wherein determining to modify the network resource is further based at least in part on a degree the second observed effect differs from the second expected effect. ### Claim 11 (depends on 7) The non-transitory computer storage medium of claim 7, wherein the process further comprises determining a rate at which a difference between the observed effect and the expected effect changes over the subsequent period of time, wherein determining to modify the network resource is further based at least partly on the rate. ### Claim 12 A system comprising: computer-readable memory storing executable instructions; and one or more processors in communication with the computer-readable memory and configured by the executable instructions to at least: determine an initial effect that a difference between a first version of a network resource and a second version of the network resource has on a metric during an initial period of time, wherein the metric relates to actions performed after accessing the network resource, and wherein the difference comprises at least one of: a portion of the first version is generated using a different algorithm than a corresponding portion of the second version, the portion of first version is presented in a different manner than the corresponding portion of the second version, or interaction with the portion of the first version causes a different action than interaction with the corresponding portion of the second version; determine a number of exposures to the network resource occurring during a subsequent period of time following the initial period of time; determine an expected effect that the difference between the first version and the second version is likely to have on the metric during the subsequent period of time, wherein the expected effect is determined based at least in part on the initial effect and the number of exposures; determine an observed effect that the difference between the first version and the second version has on the metric during the subsequent period of time, wherein the observed effect is determined based at least in part on data regarding actions performed during the subsequent period of time, and wherein an action performed by an individual user of the network resource during the subsequent period of time is performed after a user device of the individual user receives either the first version or the second version; determine to modify the network resource based at least in part on a degree the observed effect differs from the expected effect; and modify the network resource. ### Claim 13 (depends on 12) The system of claim 12, wherein the network resource comprises at least one of an audio presentation, a visual presentation, or a tactile presentation. ### Claim 14 (depends on 12) The system of claim 12, wherein the one or more processors are further configured to receive data regarding actions performed after exposure to the network resource during the initial period of time, wherein an action performed by a first user during the initial period of time is performed after a first user device of the first user receives the first version, and wherein an action performed by a second user during the initial period of time is performed after a second user device of the second user receives the second version. ### Claim 15 (depends on 12) The system of claim 12, wherein the metric relates to at least one of: selections of a portion of the first version or the second version; transactions initiated using the network resource; or actions performed after receiving the network resource. ### Claim 16 (depends on 12) The system of claim 12, wherein the one or more processors are further configured to: determine a second expected effect that the difference between the first version and the second version is likely to have on a second metric during the subsequent period of time; and determine a second observed effect that the difference between the first version and the second version has on the second metric during the subsequent period of time, wherein determining to modify the network resource is further based at least in part on a degree the second observed effect differs from the second expected effect. ### Claim 17 (depends on 12) The system of claim 12, wherein the one or more processors are further configured to determine a rate at which a difference between the observed effect and the expected effect changes over the subsequent period of time. ### Claim 18 (depends on 17) The system of claim 17, wherein the one or more processors are configured to execute the executable instructions to determine to modify the network resource based at least partly on the rate. ### Claim 19 (depends on 12) The system of claim 12, wherein the one or more processors are configured to execute the executable instructions to modify the network resource by implementing an algorithm associated with the second version. ### Claim 20 (depends on 12) The system of claim 12, wherein the one or more processors are configured to execute the executable instructions to modify the network resource by implementing a presentation change associated with the second version.