Content relevance in a social networking system using quality controlled human raters

A computer program product comprising a non-transitory computer-readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to:

• receive an indication of a test population of candidate human raters to be considered for selection into a final pool of human raters that provide ratings on different feeds of ranked content items presented in a social networking system;
• quality control testing each candidate human rater of the test population, the testing including steps conducted for each candidate human rater comprising:
• present a feed of content items to the candidate human rater,
• receive ratings of each of the content items provided by the candidate human rater,
• perform quality checks to compare at least one of the received ratings of the candidate human rater on a content item against another of the received ratings of the same candidate human rater on the same content item or on a similar content item, and
• determine a consistency score for the candidate human rater based on the performed quality checks, the consistency score representing a degree of consistency between different ones of the received ratings of the candidate human rater, the consistency score determined based on a plurality of consistency factors;
• and
• select certain of the candidate human raters from the test population for inclusion into the final pool of human raters based on the determined consistency scores from the testing performed, the final pool of human raters selected to provide ratings over a period of time on different feeds of ranked content items to improve a feed ranking model that ranks the content items for feeds presented to users of the social networking system.