Distributing a task to multiple workers over a network for completion while providing quality control

A system for selectively distributing unit jobs of a task for completion by selected workers, the system comprising:

• means for identifying a target accuracy for completion of a unit job of a task, the task including multiple unit jobs;
• means for retrieving a set of statistics for multiple workers across multiple crowdsourcing systems connected via a computer network, wherein the set of statistics include characteristics of previous responses for previous tasks performed by the multiple workers on the multiple crowdsourcing systems;
• means for setting, for each tool of the subset of tools, a wage value for completion of each unit job of the task based on the target accuracy and based on a total wage value for completion of the task;
• means for determining a number of results to be obtained for the unit job of the task based on the target accuracy, wherein said determining includes:
• means for obtaining, for each unit job of the task, a minimum number of results and a maximum number of results for completing the unit job of the task, the maximum number of results indicative of the maximum number of results capable of being performed at the wage value for completion of each unit job of the task without exceeding the total wage value for completion of the task;
• responsive to determining that a determined number of results is below the minimum number of results or greater than the maximum number of results, means for modifying, for each unit job, the number of results for the unit job of the task until the number of results both exceeds the minimum number of results and falls below the maximum number of results;
• means for obtaining a selection, for the unit job of the task, of a subset of the multiple workers across the multiple crowdsourcing systems to generate a number of responses to the unit job based on the set of statistics, wherein a number of workers included in the selected subset of the multiple workers corresponds to the determined number of results;
• means for transmitting a request over the computer network to each of the subset of the multiple workers via the multiple crowdsourcing systems, wherein each of the subset of the multiple workers are configured to perform the unit job on a corresponding crowdsourcing system of the multiple crowdsourcing systems;
• means for receiving a response from each of the subset of workers that includes the response to the unit job over the computer network via the multiple crowdsourcing systems;
• means for processing each received response to derive an accuracy value of each received response based on a similarity of each response to other received responses;
• responsive to determining that each received response from the subset of workers exceeds the target accuracy, means for distributing wage resources to the set of workers based on the set wage value associated with each of the subset of workers and the total wage value for completion of the task;
• and
• means for updating the set of statistics relating to each of the subset of workers based on the derived accuracy value of the received response provided by each of the subset of workers.

A method comprising:

• Identifying, by a server, a target accuracy for completion of a unit job of a task, the task including multiple unit jobs;
• retrieving, by the server, a set of statistics for multiple workers across multiple crowdsourcing systems connected via a computer network, wherein the set of statistics include characteristics of previous responses for previous tasks performed by the multiple workers on the multiple crowdsourcing systems;
• setting, by the server, for each tool of the subset of tools, a wage value for completion of each unit job of the task based on the target accuracy and based on a total wage value for completion of the task;
• determining, by the server, a number of results to be obtained for the unit job of the task based on the target accuracy, wherein said determining includes:
• obtaining, by the server, for each unit job of the task, a minimum number of results and a maximum number of results for completing the unit job of the task, the maximum number of results indicative of the maximum number of results capable of being performed at the wage value for completion of each unit job of the task without exceeding the total wage value for completion of the task;
• responsive to determining that a determined number of results is below the minimum number of results or greater than the maximum number of results, modifying, by the server, for each unit job, the number of results for the unit job of the task until the number of results both exceeds the minimum number of results and falls below the maximum number of results;
• obtaining, by the server, a selection, for the unit job of the task, of a subset of the multiple workers across the multiple crowdsourcing systems to generate a number of responses to the unit job based on the set of statistics, wherein a number of workers included in the selected subset of the multiple workers corresponds to the determined number of results;
• transmitting, by the server, a request over the computer network to each of the subset of the multiple workers via the multiple crowdsourcing systems, wherein each of the subset of the multiple workers are configured to perform the unit job on a corresponding crowdsourcing system of the multiple crowdsourcing systems;
• receiving, by the server, a response from each of the subset of workers that includes the response to the unit job over the computer network via the multiple crowdsourcing systems;
• processing, by the server, each received response to derive an accuracy value of each received response based on a similarity of each response to other received responses;
• and
• responsive to determining that each received response from the subset of workers exceeds the target accuracy, distributing, by the server, wage resources to the set of workers based on the set wage value associated with each of the subset of workers and the total wage value for completion of the task.

A method comprising:

• Identifying, by a processor, a target accuracy for completion of a unit job of a task, the task including multiple unit jobs;
• retrieving, by the processor, a set of statistics for multiple workers across multiple crowdsourcing systems connected via a computer network, wherein the set of statistics include characteristics of previous responses for previous tasks performed by the multiple workers on the multiple crowdsourcing systems;
• setting by the processor, for each tool of the subset of tools, a wage value for completion of each unit job of the task based on the target accuracy and based on a total wage value for completion of the task;
• determining, by the processor, a number of results to be obtained for the unit job of the task based on the target accuracy, wherein said determining includes:
• obtaining, for each unit job of the task, a minimum number of results and a maximum number of results for completing the unit job of the task, the maximum number of results indicative of the maximum number of results capable of being performed at the wage value for completion of each unit job of the task without exceeding the total wage value for completion of the task;
• responsive to determining that a determined number of results is below the minimum number of results or greater than the maximum number of results, modifying, for each unit job, the number of results for the unit job of the task until the number of results both exceeds the minimum number of results and falls below the maximum number of results;
• obtaining, by the processor, a selection, for the unit job of the task, of a subset of the multiple workers across the multiple crowdsourcing systems to generate a number of responses to the unit job based on the set of statistics, wherein a number of workers included in the selected subset of the multiple workers corresponds to the determined number of results;
• transmitting, by the processor, a request over the computer network to each of the subset of the multiple workers via the multiple crowdsourcing systems, wherein each of the subset of the multiple workers are configured to perform the unit job on a corresponding crowdsourcing system of the multiple crowdsourcing systems;
• receiving, by the processor, a response from each of the subset of workers that includes the response to the unit job over the computer network via the multiple crowdsourcing systems;
• processing, by the processor, each received response to derive an accuracy value of each received response based on a similarity of each response to other received responses;
• responsive to determining that each received response from the subset of workers exceeds the target accuracy, distributing, by the processor, wage resources to the set of workers based on the set wage value associated with each of the subset of workers and the total wage value for completion of the task;
• and
• updating, by the processor, the set of statistics relating to each of the subset of workers based on the derived accuracy value of the received response provided by each of the subset of workers.