US 2022/0076184 A1Published application

Efficient ongoing evaluation of human intelligence task contributors

AssigneeDefinedCrowd Corporation

InventorsJorge Filipe Lopes Ribeiro, Rui Pedro dos Santos Correia, João Dinis Colaço de Freitas

Life of this patent

Estimating the term…

Loading patent…

Abstract

A facility for assessing a crowdsourcing platform contributor is described. By combining a first probability value for the contributor with a source of randomness, the facility determines that a gold HIT should be presented to the contributor. In response, the facility presents a gold HIT to the contributor, and receives a response. Where the response is correct, the facility reduces the first probability value to obtain a second probability value, otherwise the facility decreases the first probability value to obtain the second probability value. The facility then determines whether a gold HIT or a regular HIT should next be presented to the contributor by combining the second probability value with a source of randomness.

Claims20

Dependent claims

A computing system, comprising:

one or more processors;
and
one or more memories collectively having contents adapted to be executed by the one or more processors to cause the instantiation and operation of a dynamic gold HIT assignment system with respect to a contributor, the dynamic gold HIT assignment system comprising:
a gold comparator configured to determine whether a gold HIT assigned to the contributor by a HIT assigner was answered correctly;
a confidence calculator configured to adjust a confidence score for the contributor based on whether the gold comparator determined that the gold HIT was answered correctly;
a suspicion calculator configured to use a suspicion kernel function to determine an updated suspicion level for the contributor from the adjusted confidence score;
the HIT assigner configured to determine, for each HIT execution to be assigned to the contributor, whether to assign a gold HIT or a regular HIT based on combining a source of randomness with the adjusted confidence score;
and
a gold score combinator configured to maintain a gold score for the contributor by aggregating the results for gold HITs assigned to the contributor determined by the gold comparator.

6 more elements

One or more computer memories collectively having contents configured to cause a computing system to perform a method with respect to a crowdsourcing platform contributor, the method comprising:

first determining, by combining a first probability value for the contributor with a source of randomness, that a gold HIT should be presented to the contributor;
in response to the first determining, presenting a gold HIT to the contributor;
receiving a response to the presented gold HIT;
where the received response is correct, reducing the first probability value to obtain a second probability value;
where the received response is incorrect, increasing the first probability value to obtain the second probability value;
and
second determining, by combining the second probability value with a source of randomness, whether a gold HIT or a regular HIT should next be presented to the contributor.

4 more elements

Dependencies are inferred from the claim text.