# US 9,519,916 B2 · System and method for concurrently conducting cause-and-effect experiments on content effectiveness and adjusting content distribution to optimize business objectives > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,519,916 B2 - Document ID: US9519916B2 - Status: Granted - Kind: B2 - Application number: 13/875,917 - Filed: 2013-05-02 - Published: 2013-09-19 - Granted: 2016-12-13 - Earliest claimed: 2009-01-07 - Assignees: 3M Innovative Properties Company - Applicants: 3M Innovative Properties Company - Inventors: Brian E. Brooks; Brian J. Stankiewicz; Jonathan B. Arthur; Craig G. Markell; Brian L. Linzie - Web page: https://patentgrep.com/patent/US9519916B2 ## CPC - G06Q 30/0244: {Optimization} - G06N 20/00: Machine learning - G06Q 10/04: Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" (market predictions or forecasting for commercial activities G06Q30/0202) - G06Q 30/02: Marketing; Price estimation or determination; Fundraising - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract The present invention is directed to systems, articles, and computer-implemented methods for assessing effectiveness of communication content and optimizing content distribution to enhance business objectives. Embodiments of the present invention are directed to computer-implemented methods for a computer-implemented method, comprising conducting an experiment using experimental content to determine effectiveness of communication content and executing, while conducting the experiment, a machine learning routine (MLR) using MLR content to enhance an effectiveness metric. ## Claims ### Claim 1 A computer system having one or more processors and memories, comprising: an experiment module, executed on the one or more processors, configured to conduct an experiment, and use experimental content to determine effectiveness of communication content in accordance with a schedule comprising a first set and a second set of time periods, the experiment conducted using the first set of time periods; and a machine learning routine (MLR) module, executed on the one or more processors, configured to execute, while conducting the experiment, a machine learning routine, and use MLR content to enhance an effectiveness metric in accordance with the schedule, wherein: the machine learning routine comprises one or both of an explore routine associated with explore content and an exploit routine associated with exploit content; for experimental content that are related to either of explore or exploit content, using the second set of time periods for the explore or exploit routine; and for experimental content that are unrelated to either of explore or exploit content, using at least some of the first set of time periods for the explore or exploit routine. ### Claim 2 (depends on 1) The computer of claim 1, further comprising: a module for displaying the experimental content and the MLR content on a display according to the schedule. ### Claim 3 (depends on 1) The computer system of claim 1, wherein the machine learning routine comprises a reinforcement learning routine, a logistic regression routine, an unsupervised learning routine, a semi-supervised learning routine, or use of one or more neural networks. ### Claim 4 (depends on 1) The computer system of claim 1, wherein the MLR module is operative to identify a frequency rate for presenting the explore content and the exploit content that results in enhancement of the predetermined effectiveness metric. ### Claim 5 (depends on 1) The computer system of claim 1, further comprising: an evaluation module, executed on the one or more processors, configured to perform an evaluation to determine, for any given time period, if using experimental content has more value than using MLR content for the time period. ### Claim 6 (depends on 1) The computer system of claim 1, further comprising: a schedule module, executed on the one or more processors, configured to generate the schedule, wherein the module receives one or more of (a) a value of the experimental hypothesis or hypotheses, (b) a value of each category of effectiveness metric, (c) viewer visit duration (VVD) for each business objective, (d) content restrictions, and (e) urgency information, and wherein the schedule module generates the schedule in accordance with one or more of (a) through (e). ### Claim 7 (depends on 6) The computer system of claim 6, wherein the schedule module continuously adjusts the schedule in terms of times and display locations to maximize at least one effectiveness metric. ### Claim 8 (depends on 1) The computer system of claim 1, further comprising: an analysis module, executed on the one or more processors, configured to analyze measurement data indicative of MLR content effectiveness for the machine learning routine, and store the measurement data in a historical database. ### Claim 9 (depends on 1) The computer system of claim 1, further comprising: an output module, executed on the one or more processors, configured to produce output data indicative of return on investment for each piece of MLR content based on one or both of daypart and display location. ### Claim 10 A computer-implemented method comprising: conducting, by a processing system, an experiment using experimental content to determine effectiveness of communication content in accordance with a schedule comprising a first set and a second set of time periods, the experiment conducted using the first set of time periods; and executing, by the processing system, while conducting the experiment, a machine learning routine (MLR) using MLR content to enhance an effectiveness metric in accordance with the schedule, wherein: the machine learning routine comprises one or both of an explore routine associated with explore content and an exploit routine associated with exploit content; for experimental content that are related to either of explore or exploit content, using the second set of time periods for the explore or exploit routine; and for experimental content that are unrelated to either of explore or exploit content, using at least some of the first set of time periods for the explore or exploit routine. ### Claim 11 (depends on 10) The computer-implemented method of claim 10, further comprising: displaying experimental content and MLR content on a display according to the schedule. ### Claim 12 (depends on 10) The computer-implemented method of claim 10, wherein the machine learning routine comprises a reinforcement learning routine, a logistic regression routine, an unsupervised learning routine, a semi-supervised learning routine, or use of one or more neural networks. ### Claim 13 (depends on 10) The computer-implemented method of claim 10, further comprising identifying a frequency rate for presenting the explore content and the exploit content that results in enhancement of the predetermined effectiveness metric. ### Claim 14 (depends on 10) The computer-implemented method of claim 10, further comprising: performing, by the processing system, an evaluation to determine, for any given time period, if using experimental content has more value than using MLR content for the time period. ### Claim 15 (depends on 10) The computer-implemented method of claim 10, further comprising: receiving one or more of (a) a value of the experimental hypothesis or hypotheses, (b) a value of each category of effectiveness metric, (c) viewer visit duration (VVD) for each business objective, (d) content restrictions, and (e) urgency information; and generating, by the processing system, the schedule for implementing the experiment and machine learning routine in accordance with one or more of (a) through (e). ### Claim 16 (depends on 15) The computer-implemented method of claim 15, further comprising: continuously adjusting the schedule in terms of times and display locations to maximize at least one effectiveness metric. ### Claim 17 (depends on 10) The computer-implemented method of claim 10, further comprising: analyzing, by the processing system, measurement data indicative of MLR content effectiveness for the machine learning routine, and storing the measurement data in a historical database. ### Claim 18 (depends on 10) The computer-implemented method of claim 10, further comprising producing output data indicative of return on investment for each piece of MLR content based on one or both of daypart and display location. ### Claim 19 A computer system having one or more processors and memories, comprising: a module executed on the one or more processors configured to conduct an experiment and use experimental content to determine effectiveness of communication content in accordance with a schedule comprising a first plurality of time-slot samples and a second plurality of time-slot samples, the experiment being conducted using the first plurality of time-slot samples; and a module executed on the one or more processors configured to execute, while conducting the experiment, a machine learning routine (MLR) and use MLR content to enhance an effectiveness metric in accordance with the schedule, wherein: the machine learning routine comprises a reinforcement learning routine, the reinforcement learning routine comprising one or both of an explore routine associated with explore content and an exploit routine associated with exploit content; for experimental content that are related to either of explore or exploit content, using the second plurality of the time-slot samples for the explore or exploit routine; and for experimental content that are unrelated to either of explore or exploit content, using at least some of the first plurality of time-slot samples for the explore or exploit content.