# US 10,504,136 B2 · Measuring performance of content among groups of similar users of an online system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,504,136 B2 - Document ID: US10504136B2 - Status: Granted - Kind: B2 - Application number: 14/997,465 - Filed: 2016-01-15 - Published: 2017-07-20 - Granted: 2019-12-10 - Earliest claimed: 2016-01-15 - Assignees: Facebook, Inc. - Applicants: Facebook, Inc. - Inventors: Patrick James Kemp; Frederick Ross Leach - Web page: https://patentgrep.com/patent/US10504136B2 ## CPC - G06Q 30/0242: Determining effectiveness of advertisements - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR ## Abstract An online system generates metrics describing performance of a content item presented to its users by various entities based on a number of events associated with an exposed group of users presented with a content item and a control group of users not presented with the content item and similar to users of the exposed group. The online system receives information describing presentation of the content item to each user of the exposed group via client devices associated with users of the exposed group. The online system provides information used by the online system to identify each user in the exposed group and in the control group to a trusted third party, which identifies information describing events associated with the content item from one or more entities. Based on the identified information, a metric is generated to describe the effectiveness of the content item. ## Claims ### Claim 1 A method comprising: storing, by a social networking server of an online system, an edge store storing connections between users of the online system; receiving, at the online system, information describing a content item presented to each user of an exposed group of users of the online system via a client device associated with each user in the exposed group, the information including a time of presentation of the content item to each user in the exposed group; obtaining user identifying information from the client device associated with each user of the exposed group of users; identifying a non-exposed group of users of the online system outside of the exposed group of users; generating a measure of similarity between the exposed group of users and each of the non-exposed group of users based on stored attributes including the stored connections between users in the edge store of the social networking server; determining information identifying a control group of users as a subset of the non-exposed group of users having at least a threshold measure of similarity to one or more users of the exposed group of users; generating hashes of the user identifying information for the exposed group of users and the information identifying the control group of users to obfuscate identities of the exposed group of users and the control group of users; providing the hashes to a third party system, the third party system: accessing information describing a number of occurrences of an event associated with the content item and information identifying each of a plurality of online system users associated with the event, determining a number of occurrences of the event associated with the users of the exposed group of users and a number of occurrences of the event associated with the users of the control group of users based at least in part on the hashes and the accessed information, and generating a metric describing the number of occurrences of the event associated with the exposed group of users and the number of occurrences of the event associated with the control group of users, wherein the metric is based at least in part on a difference between the number of occurrences of the event associated with the users of the exposed group of users and the number of occurrences of the event associated with the users of the control group of users. ### Claim 2 (depends on 1) The method of claim 1, wherein the metric is based at least in part on a difference between the number of occurrences of the event associated with the users of the exposed group of users and the number of occurrences of the event associated with the users of the control group of users. ### Claim 3 (depends on 1) The method of claim 1, wherein the content item comprises an advertisement. ### Claim 4 (depends on 1) The method of claim 1, wherein each user of the control group of users satisfies one or more targeting criteria associated with the content item. ### Claim 5 (depends on 1) The method of claim 1, wherein the threshold measure of similarity is based at least in part on a threshold number of attributes associated with a user of the control group matching attributes associated with users of the exposed group of users. ### Claim 6 (depends on 5) The method of claim 5, wherein the attribute associated with the user of the control group of users is selected from a group consisting of: an action performed by the user of the control group of users on the online system, an action performed by the user of the control group of users on a third party system, a connection between the user of the control group of users and an additional user of the online system, a connection between the user of the control group of users and an object on the online system, and any combination thereof. ### Claim 7 (depends on 5) The method of claim 5, wherein the attribute associated with the user of the control group of users is selected from a group consisting of: an age of the user of the control group of users, a gender of the user of the control group of users, a geographic area associated with the user of the control group of users, and any combination thereof. ### Claim 8 (depends on 1) The method of claim 1, wherein identifying the control group of users having at least the threshold measure of similarity to the users of the exposed group of users comprises: comparing attributes of each of a plurality of users user who were not presented with the content item within at least the threshold interval of time from the time of presentation of the content item to one or more users of the exposed group of users to attributes of one or more users of the exposed group of users; computing a similarity score for each user of the plurality of users, the similarity score for a user of the plurality of users based on the comparison of attributes of the plurality of the users to attributes of one or more users of the exposed group of users; and identifying, the control group of users as users of the plurality of users having at least a similarity score equaling or exceeding the threshold measure of similarity. ### Claim 9 (depends on 8) The method of claim 8, wherein an attribute associated with one or more users of the exposed group of users comprises an attribute shared by at least a threshold number of the users of the exposed group of users. ### Claim 10 (depends on 1) The method of claim 1, wherein the event is selected from a group consisting of: a location of a user of the online system being within a threshold distance of a location associated with the content item, the user of the online system accessing a website associated with a content item, the user of the online system accessing the content item, the user of the online system indicating a preference for the content item, the user of the online system downloading an application associated with the content item, the user of the online system purchasing a product or a service associated with the content item, the user accessing an object maintained by the online system and associated with the content item, and any combination thereof. ### Claim 11 (depends on 1) The method of claim 1, wherein the information describing the number of occurrences of the event associated with the content item and information identifying each of the plurality of online system users associated with the event comprises information received from one or more entities. ### Claim 12 (depends on 1) The method of claim 1, wherein determining the number of occurrences of the event associated with the users of the exposed group of users and the number of occurrences of the event associated with the users of the control group of users comprises: retrieving information correlating information identifying users of the online system with information identifying users to an entity that provided at least a set of the accessed information; determining information identifying users of the online system correlated with information identifying users to the entity based on the retrieved information; determining the number of occurrences of the event associated with the users of the exposed group of users based at least in part on a number of occurrences of the event associated with information identifying users to the entity correlated with information identifying users of the exposed group to the online system; and determining the number of occurrences of the event associated with the users of the control group of users based at least in part on a number of occurrences of the event associated with information identifying users to the entity correlated with information identifying users of the control group to the online system. ### Claim 13 A method comprising: storing, by a social networking server of an online system, an edge store storing connections between users of the online system; receiving, at the online system, information describing a content item presented to each user of an exposed group of users of the online system via a client device associated with each user in the exposed group, the information including a time of presentation of the content item to each user in the exposed group; obtaining user identifying information from the client device associated with each user of the exposed group of users; identifying a non-exposed group of users of the online system outside of the exposed group of users; generating a measure of similarity between the exposed group of users and each of the non-exposed group of users based on stored attributes including the stored connections between users in the edge store of the social networking server; determining information identifying a control group of users as a subset of the non-exposed group of users having at least a threshold measure of similarity to one or more users of the exposed group of users; generating hashes of the user identifying information for the exposed group of users and the information identifying the control group of users to obfuscate identities of the exposed group of users and the control group of users; providing the hashes to a third party system, wherein the third party system generates a metric describing a number of occurrences of an event associated with the exposed group of users and a number of occurrences of the event associated with the control group of users based on the hashes, wherein the metric is based at least in part on a difference between the number of occurrences of the event associated with the users of the exposed group of users and the number of occurrences of the event associated with the users of the control group of users. ### Claim 14 (depends on 13) The method of claim 13, wherein the content item comprises an advertisement. ### Claim 15 (depends on 13) The method of claim 13, wherein each user of the control group of users satisfies one or more targeting criteria associated with the content item. ### Claim 16 (depends on 13) The method of claim 13, wherein the threshold measure of similarity is based at least in part on a threshold number of attributes associated with a user of the control group matching attributes associated with users of the exposed group of users. ### Claim 17 (depends on 16) The method of claim 16, wherein the attribute associated with the user of the control group of users is selected from a group consisting of: an action performed by the user of the control group of users on the online system, an action performed by the user of the control group of users on a third party system, a connection between the user of the control group of users and an additional user of the online system, a connection between the user of the control group of users and an object on the online system, and any combination thereof. ### Claim 18 (depends on 16) The method of claim 16, wherein the attribute associated with the user of the control group of users is selected from a group consisting of: an age of the user of the control group of users, a gender of the user of the control group of users, a geographic area associated with the user of the control group of users, and any combination thereof. ### Claim 19 (depends on 13) The method of claim 13, wherein identifying the control group of users having at least the threshold measure of similarity to the users of the exposed group of users comprises: comparing attributes of each of a plurality of users user who were not presented with the content item within at least the threshold interval of time from the time of presentation of the content item to one or more users of the exposed group of users to attributes of one or more users of the exposed group of users; computing a similarity score for each user of the plurality of users, the similarity score for a user of the plurality of users based on the comparison of attributes of the plurality of the users to attributes of one or more users of the exposed group of users; and identifying, the control group of users as users of the plurality of users having at least a similarity score equaling or exceeding the threshold measure of similarity. ### Claim 20 (depends on 19) The method of claim 19, wherein an attribute associated with one or more users of the exposed group of users comprises an attribute shared by at least a threshold number of the users of the exposed group of users. ### Claim 21 (depends on 13) The method of claim 13, wherein the third party system accesses information describing the number of occurrences of the event associated with the content item and information identifying each of a plurality of online system users associated with the event, and determines the number of occurrences of the event associated with the users of the exposed group of users and the number of occurrences of the event associated with the users of the control group of users based at least in part on a comparison of the provided information and the accessed information. ### Claim 22 (depends on 21) The method of claim 21, wherein the metric is based at least in part on a difference between the determined number of occurrences of the event associated with the users of the exposed group of users and the determined number of occurrences of the event associated with the users of the control group of users. ### Claim 23 A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to: storing, by a social networking server of an online system, an edge store storing connections between users of the online system; receiving, at the online system, information describing a content item presented to each user of an exposed group of users of the online system via a client device associated with each user in the exposed group, the information including a time of presentation of the content item to each user in the exposed group; obtaining user identifying information from the client device associated with each user of the exposed group of users; identifying a non-exposed group of users of the online system outside of the exposed group of users; generating a measure of similarity between the exposed group of users and each of the non-exposed group of users based on stored attributes including the stored connections between users in the edge store of the social networking server; determining information identifying a control group of users as a subset of the non-exposed group of users having at least a threshold measure of similarity to one or more users of the exposed group of users; generating hashes of the user identifying information for the exposed group of users and the information identifying the control group of users to obfuscate identities of the exposed group of users and the control group of users; providing the hashes to a third party system, the third party system: accessing information describing a number of occurrences of an event associated with the content item and information identifying each of a plurality of online system users associated with the event, determining a number of occurrences of the event associated with the users of the exposed group of users and a number of occurrences of the event associated with the users of the control group of users based at least in part on the hashes and the accessed information, and generating a metric describing the number of occurrences of the event associated with the exposed group of users and the number of occurrences of the event associated with the control group of users, wherein the metric is based at least in part on a difference between the number of occurrences of the event associated with the users of the exposed group of users and the number of occurrences of the event associated with the users of the control group of users.