# US 11,907,968 B1 · Media effectiveness > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,968 B1 - Document ID: US11907968B1 - Status: Granted - Kind: B1 - Application number: 17/983,946 - Filed: 2022-11-09 - Granted: 2024-02-20 - Earliest claimed: 2016-10-04 - Assignees: United Services Automobile Association (USAA) - Applicants: United Services Automobile Association (USAA) - Inventors: Richmond E. Meier; Charles Holbrook Moore; Douglas Michael Mowen - Web page: https://patentgrep.com/patent/US11907968B1 ## CPC - G06Q 30/0243: {Comparative campaigns} - G06Q 30/0244: {Optimization} - G06Q 30/0277: {Online advertisement} - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR ## Abstract Methods, systems, and apparatus, including computer programs encoded on computer storage media, to improve the impact of digital media. One of these methods includes providing a non-affiliated advertisement to a plurality of users. The method includes receiving indications that a subset of the plurality of users have viewed the non-affiliated advertisement. The method also includes creating a control group for an advertisement from the subset of the plurality of users that received the non-affiliated advertisement. ## Claims ### Claim 1 A computer-implemented method of creating a control group for determining effectiveness of a target advertisement, the method comprising: transmitting, by at least one processor, advertisement data representing a non-affiliated advertisement to a plurality of user devices; identifying one or more users of one or more user devices who did not view the non-affiliated advertisement; generating, by at least one processor, control group data for a target advertisement, the control group data comprising data representing the one or more user devices being assigned to a test group; transmitting, by at least one processor and to the one or more user devices assigned to the test group, the target advertisement; monitoring transactions of users whose devices are assigned to the test group and users whose devices are not assigned to the test group to determine an average conversion rate of the users whose devices are assigned to the test group and of the users whose devices are not assigned to the test group; and computing a lift value based on a percentage difference between the average conversion rate of the users whose devices are not assigned to the test group and the average conversion rate of the users whose devices are assigned to the test group. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the advertisement data comprises a callback function that, when executed by a user device of the plurality of user devices that is configured to receive the advertisement data, causes that user device to transmit, to at least one processor, callback data indicating that a user of that user device has viewed the non-affiliated advertisement and comprising identification data of that corresponding user. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein the identification data comprises a cookie. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein transmitting the advertisement data comprises transmitting the advertisement data to an advertising system. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the non-affiliated advertisement is a public service message or an advertisement for a service unrelated to a service promoted in the target advertisement. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the advertisement data representing the non-affiliated advertisement is transmitted in response to a request for the advertisement data from each of the plurality of user devices. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, further comprising adjusting a marketing campaign based on the lift value. ### Claim 8 (depends on 7) The computer-implemented method of claim 7, wherein adjusting a marketing campaign comprises performing an analysis to determine whether revenue generated due, at least in part, to the non-affiliated advertisement provides a desired return on investment compared to costs associated with the non-affiliated advertisement. ### Claim 9 A non-transitory computer storage medium encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations for determining effectiveness of a target advertisement, the operations comprising: transmitting advertisement data representing a non-affiliated advertisement to a plurality of user devices; identifying one or more users of one or more user devices who did not view the non-affiliated advertisement; generating control group data for a target advertisement, the control group data comprising data representing the one or more user devices being assigned to a test group; transmitting the target advertisement to the one or more user devices assigned to the test group; monitoring transactions of users whose devices are assigned to the test group and users whose devices are not assigned to the test group to determine an average conversion rate of the users whose devices are assigned to the test group and of the users whose devices are not assigned to the test group; and computing a lift value based on a percentage difference between the average conversion rate of the users whose devices are not assigned to the test group and the average conversion rate of the users whose devices are assigned to the test group. ### Claim 10 (depends on 9) The non-transitory computer storage medium of claim 9, wherein the advertisement data comprises a callback function that, when executed by a user device of the plurality of user devices that is configured to receive the advertisement data, causes that user device to transmit callback data indicating that a user of that user device has viewed the non-affiliated advertisement and comprising identification data of that corresponding user. ### Claim 11 (depends on 10) The non-transitory computer storage medium of claim 10, wherein the identification data comprises a cookie. ### Claim 12 (depends on 9) The non-transitory computer storage medium of claim 9, wherein the non-affiliated advertisement is a public service message or an advertisement for a service unrelated to a service promoted in the target advertisement. ### Claim 13 (depends on 9) The non-transitory computer storage medium of claim 9, wherein the operations further comprise adjusting a marketing campaign based on the lift value. ### Claim 14 A system comprising: one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for determining effectiveness of a target advertisement, the operations comprising: transmitting advertisement data representing a non-affiliated advertisement to a plurality of user devices; identifying one or more users of one or more user devices who did not view the non-affiliated advertisement; generating control group data for a target advertisement, the control group data comprising data representing the one or more user devices being assigned to a test group; transmitting the target advertisement to the one or more user devices assigned to the test group; monitoring transactions of users whose devices are assigned to the test group and users whose devices are not assigned to the test group to determine an average conversion rate of the users whose devices are assigned to the test group and of the users whose devices are not assigned to the test group; and computing a lift value based on a percentage difference between the average conversion rate of the users whose devices are not assigned to the test group and the average conversion rate of the users whose devices are assigned to the test group. ### Claim 15 (depends on 14) The system of claim 14, wherein the advertisement data comprises a callback function that, when executed by a user device of the plurality of user devices that is configured to receive the advertisement data, causes that user device to transmit, to at least one processor, callback data indicating that a user of that user device has viewed the non-affiliated advertisement and comprising identification data of that corresponding user. ### Claim 16 (depends on 15) The system of claim 15, wherein the identification data comprises a cookie. ### Claim 17 (depends on 14) The system of claim 14, wherein the non-affiliated advertisement is a public service message or an advertisement for a service unrelated to a service promoted in the target advertisement. ### Claim 18 (depends on 14) The system of claim 14, wherein the advertisement data representing the non-affiliated advertisement is transmitted in response to a request for the advertisement data from each of the plurality of user devices. ### Claim 19 (depends on 14) The system of claim 14, wherein the operations further comprise adjusting a marketing campaign based on the lift value. ### Claim 20 (depends on 19) The system of claim 19, wherein adjusting a marketing campaign comprises performing an analysis to determine whether revenue generated due, at least in part, to the non-affiliated advertisement provides a desired return on investment compared to costs associated with the non-affiliated advertisement.