Content relevance in a social networking system using population-representative human rater pool

A computer-implemented method comprising:

• identifying a pool of human raters for providing ratings on feeds of content items of a social networking system, the pool of human raters identified to represent a user population of interest to the social networking system, wherein the social networking system comprises a computer system;
• providing, by the social networking system for display to each rater of the identified pool of human raters, a first feed of content items;
• obtaining, from each rater of the identified pool of human raters, ratings of each of a plurality of content items of the first feed;
• removing one or more human raters from the identified pool of human raters based on performing quality checks on the obtained ratings data;
• determining a representativeness score for the pool of human raters after the one or more human raters are removed based on the obtained ratings data, the representativeness score indicating a level of similarity between a composition of the pool of human raters and a composition of the user population of interest after the one or more human raters are removed, the representativeness score determined based on a plurality of representativeness factors;
• revising the composition of the pool of human raters based on the determined representativeness score of the pool of human raters, the revised pool of human raters representing the user population of interest;
• providing, by the social networking system for display to each rater of the revised pool of human raters, a second feed of content items;
• obtaining, from each rater of the revised pool of human raters, ratings of each of a plurality of the content items of the second feed;
• selecting content items for users in the user population of interest based on the obtained ratings;
• and
• providing the selected content items for display to the users in the user population of interest in feeds of the social networking system.

A computer program product comprising a non-transitory computer-readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to:

• identify a pool of human raters for providing ratings on feeds of content items of a social networking system, the pool of human raters identified to represent a user population of interest to the social networking system;
• provide, by the social networking system for display to each rater of the identified pool of human raters, a first feed of content items;
• obtain, from each rater of the identified pool of human raters, ratings of each of a plurality of content items of the first feed;
• determine a representativeness score for the identified pool of human raters based on the obtained ratings data, the representativeness score indicating a level of similarity between a composition of the pool of human raters and a composition of the user population of interest, the representativeness score determined based on a plurality of representativeness factors;
• revise the composition of the pool of human raters based on the determined representativeness score of the pool of human raters, the revised pool of human raters having characteristics that are representative of characteristics of the user population of interest;
• provide, by the social networking system for display to each rater of the revised pool of human raters, a second feed of content items;
• obtain, from each rater of the revised pool of human raters, ratings of each of a plurality of the content items of the second feed;
• select content items for users in the user population of interest based on the obtained ratings;
• and
• provide the selected content items for display to the users in the user population of interest in feeds of the social networking system.