Identifying workers in a crowdsourcing or microtasking platform who perform low-quality work and/or are really automated bots

A computer-readable storage medium whose contents are capable of causing a computing system to perform a method for predicting a level of quality of work of crowdsourced workers of a micro-tasking platform, the method comprising:

• for each crowdsourced worker of a first set of crowdsourced workers in the micro-tasking platform:
• assigning tasks to the crowdsourced worker, wherein the tasks are associated with a job session;
• measuring a level of quality of work produced by the crowdsourced worker on the assigned tasks;
• observing interactions by the crowdsourced worker with the micro-tasking platform;
• and
• training a model designed to predict a level of quality of work produced by the crowdsourced worker based on the crowdsourced worker's interactions with the micro-tasking platform, using observed interactions each corresponding to a different crowdsourced worker from the first set of crowdsourced workers, in which each of the observed interactions is an independent variable and the measured level of quality of work is a dependent variable;
• observing interactions by a distinguished worker with the micro-tasking platform;
• and
• applying the trained model to the observed interactions by the distinguished worker to predict a level of quality of work that will be produced by the distinguished worker.

A method in a computing system for measuring levels of quality of work produced by crowdsourced workers of a micro-tasking platform, comprising:

• identifying a set of gold tasks associated with a job session, wherein each gold task in the set of gold tasks is associated with a gold result, and wherein the set of gold tasks comprises a set of prior gold tasks used before the associated job session and a set of present gold tasks used during the associated job session;
• computing a qualification tests score based on the set of prior gold tasks, wherein the qualification tests score is a function of a size of the set of prior gold tasks and a total number of correct answers whose result match a corresponding gold result of gold tasks in the set of prior gold tasks;
• computing a real-time audits score based on the set of present gold tasks, wherein the real-time audit score is a function of a size of the set of present gold tasks and a total number of correct answers whose result match a corresponding gold result of gold tasks in the set of present gold tasks;
• and
• computing a gold score based on the computed qualification tests score and the computed real-time audits score.