Methods and systems for designing of tasks for crowdsourcing

An image-analysis task system, the system comprising:

• an application server comprising one or more micro-processors;
• a requester computing device comprising a display device configured to display a Graphical User Interface (GUI);
• a worker computing device;
• and
• a network configured for communication of the requester computing device and the worker computing device with the application server;
• wherein:
• the application server is configured to generate the GUI that displays output parameters including a task accuracy, a task completion time, and a task acceptance rate associated with an image-analysis task on the display device;
• the application server is configured to receive, through the Graphical User Interface (GUI), first target values of the output parameters associated with the image-analysis task, wherein the first target values include values for the task accuracy, the task completion time, and the task acceptance rate;
• the application server is configured to determine first values of input parameters associated with performing the image-analysis task by performing an optimization process that is based on maximizing a mean value of a lognormal of the task accuracy, maximizing an inverse of a mean value of a lognormal of the task completion time, and maximizing the task acceptance rate associated with the image-analysis task, based at least on the first target values and a model, wherein the model corresponds to a relationship between the input parameters and the output parameters;
• the application server is configured to present through the GUI, a plurality of first solutions of the optimization process and the first values of the input parameters associated with the plurality of first solutions;
• the application server is configured to receive, through the GUI, second target values of the output parameters, the second target values replacing the first target values;
• the application server is configured to determine second values of the input parameters by performing the optimization process based on the model and the second target values;
• the application server is configured to present through the GUI, a plurality of second solutions of the optimization process and the second values of the input parameters associated with the plurality of second solutions;
• the application server is configured to receive, through the GUI, a first input selecting a second solution from the plurality of second solutions and a second input accepting the second values of the input parameters;
• and
• the application server is configured to transmit the image-analysis task and the second values of the input parameters to the worker computing device of one or more crowdworkers to enable the one or more crowdworkers to perform the image-analysis task according to the second values of the input parameters,
• wherein the first values of the input parameters are determined based at least on the first target values and the model, the second values of the input parameters are determined based at least on the second target values and the model, and the image-analysis task relates to at least one of image labelling, image tagging, image categorisation, video labelling, video tagging, video categorisation, handwriting recognition, identifying vulgar content, removing vulgar content, identifying illegal content, removing illegal content, duplicate checking, video transcription, and targeted photography of product placement.

A method for operating an image-analysis task system, the method comprising:

• generating, using an application server comprising one or more microprocessors, a Graphical User Interface (GUI) displaying output parameters including a task accuracy, a task completion time, and a task acceptance rate associated with the image-analysis task on a display device of a requester computing device;
• receiving, using the application server, first target values of the output parameters associated with the image-analysis task through the GUI, wherein the first target values include values for the task accuracy, the task completion time, and the task acceptance rate;
• determining, using the application server, first values of input parameters associated with performing the image-analysis task by performing an optimization process that is based on maximizing a mean value of a lognormal of the task accuracy, maximizing an inverse of a mean value of a lognormal of the task completion time, and maximizing the task acceptance rate associated with the image-analysis task, based at least on the first target values and a model, wherein the model corresponds to a relationship between the input parameters and the output parameters;
• displaying, through the GUI, a plurality of first solutions of the optimization process and the first values of the input parameters associated with the plurality of first solutions;
• receiving by the application server, through the GUI, second target values of the output parameters, the second target values replacing the first target values;
• determining, using the application server, second values of the input parameters by performing the optimization process based on the model and the second target values;
• displaying through the GUI, a plurality of second solutions of the optimization process and the second values of the input parameters associated with the plurality of second solutions;
• receiving by the application server, through the GUI, a first input selecting a second solution from the plurality of second solutions and a second input accepting the second values of the input parameters;
• and
• transmitting, using the application server, the image-analysis task and the second values of the input parameters to a worker computing device of one or more crowdworkers to enable the one or more crowdworkers to perform the image-analysis task according to the second values of the input parameters, the transmission occurring through a network configured for communication of the requester computing device and the worker computing device with the application server.

A method for operating one or more image-analysis tasks, the method comprising:

• generating, using an application server comprising one or more microprocessors, a Graphical User Interface (GUI) displaying output parameters including a task accuracy, a task completion time, and a task acceptance rate associated with the one or more image-analysis tasks on a display device of a requester computing device:
• receiving, using the application server, first target values of the output parameters associated with the one or more image-analysis tasks from the requester computing device through the GUI, wherein the first target values include values for the task accuracy, the task completion time, and the task acceptance rate;
• determining, using the application server, first values of input parameters associated with performing the one or more image-analysis tasks by performing an optimization process that is based on maximizing a mean value of a lognormal of the task accuracy, maximizing an inverse of a mean value of a lognormal of the task completion time, and maximizing the task acceptance rate associated with the one or more image-analysis tasks, based at least on the first target values and a model, wherein the model corresponds to a relationship between the input parameters associated with the one or more image-analysis tasks and the output parameters;
• displaying, through the GUI, a plurality of first solutions of the optimization process and the first values of the input parameters within a graphical plot, wherein the graphical plot corresponds to at least one of a Pareto Front plot or a Parallel co-ordinates plot;
• receiving, through the GUI, second target values for the output parameters, the second target values replacing the first target values;
• determining, using the application server, second values of the input parameters by performing the optimization process based on the model and the second target values;
• displaying, through the GUI, a plurality of second solutions of the optimization process and the second values of the input parameters associated with the plurality of second solutions;
• receiving, through the GUI, a first input from a requester selecting a second solution from the plurality of second solutions and a second input from the requester accepting the second values of the input parameters;
• and
• transmitting, using the application server, the one or more image-analysis tasks and the second values of the input parameters to a worker computing device of one or more crowdworkers to enable the one or more crowdworkers to perform the one or more image-analysis tasks according to the second values of the input parameters, and the one or more image-analysis tasks relate to at least one of image labelling, image tagging, image categorisation, video labelling, video tagging, video categorisation, handwriting recognition, identifying vulgar content, removing vulgar content, identifying illegal content, removing illegal content, duplicate checking, video transcription, and targeted photography of product placement, wherein the transmission occurs through a network configured for communication of the requester computing device and the worker computing device with the application server.

A computer program product for use with an image-analysis system,

• the computer program product comprising a non-transitory computer readable medium, wherein the non-transitory computer readable medium stores a computer program code for designing an image-analysis task for crowdsourcing, wherein the computer program code is executable by one or more micro-processors on an application server to:
• generate a Graphical User Interface (GUI) that displays output parameters including a task accuracy, a task completion time, and a task acceptance rate associated with the image-analysis task on a display device of a requester computing device;
• receive first target values of the output parameters associated with the image-analysis task from the requester computing device through the GUI, wherein the first target values include values for the task accuracy, the task completion time, and the task acceptance rate;
• determine first values of input parameters associated with performing the image-analysis task by performing an optimization process that is based on maximizing a mean value of a lognormal of the task accuracy, maximizing an inverse of a mean value of a lognormal of the task completion time, and maximizing the task acceptance rate associated with the image-analysis task, based at least on the first target values and a model, wherein the model corresponds to a relationship between the input parameters and the output parameters;
• display, through the GUI, a plurality of first solutions of the optimization process and the first values of the input parameters associated with the plurality of first solutions;
• receive, through the GUI, second target values of the output parameters, the second target values replacing the first target values;
• determine second values of the input parameters by performing the optimization process based on the model and the second target values;
• display, through the GUI, a plurality of second solutions of the optimization process and the second values of the input parameters associated with the plurality of second solutions;
• receive, through the GUI, a first input selecting a second solution from the plurality of second solutions and a second input accepting the second values of the input parameters;
• and
• transmit the image-analysis task and the second values of the input parameters to a worker computing device of one or more crowdworkers to enable the one or more crowdworkers to perform the image-analysis task according to the second values of the input parameters, the transmission occurring through a network configured for communication of the requester computing device and the worker computing device with the application server.

A computer program product for use with an image-analysis system,

• the computer program product comprising a non-transitory computer readable medium, wherein the non-transitory computer readable medium stores a computer program code for designing one or more image-analysis tasks for crowdsourcing, wherein the computer program code is executable by one or more micro-processors on an application server to:
• generate a Graphical User Interface (GUI) that displays output parameters including a task accuracy, a task completion time, and a task acceptance rate associated with the one or more image-analysis tasks on a display device of a requester computing device;
• receive through the GUI, first target values of the output parameters associated with the one or more image-analysis tasks, wherein the first target values include values for the task accuracy, the task completion time, and the task acceptance rate;
• determine first values of input parameters associated with performing the one or more image-analysis tasks by performing an optimization process that is based on maximizing a mean value of a lognormal of the task accuracy, maximizing an inverse of a mean value of a lognormal of the task completion time, and maximizing the task acceptance rate, the optimization process is performed based on the first target values and a model representing a relationship between the input parameters and the output parameters;
• present, through the GUI, a plurality of first solutions of the optimization process and the first values of the input parameters associated with the plurality of first solutions;
• receive, through the GUI, second target values of the output parameters, the second target values replacing the first target values;
• determine second values of the input parameters by performing the optimization process based on the model and the second target values;
• present, through the GUI, a plurality of second solutions of the optimization process and the second values of the input parameters associated with the plurality of second solutions;
• receive, through the GUI, a first input selecting a second solution from the plurality of second solutions and a second input accepting the second values of the input parameters;
• and
• transmit the one or more image-analysis tasks and the second values of the input parameters to a worker computing device of one or more crowdworkers to enable the one or more crowdworkers to perform the one or more image-analysis tasks according to the second values of the input parameters,
• wherein the first values of the input parameters are determined based at least on the first target values and the model, the second values of the input parameters are determined based on the second target values and the model, and the image-analysis task relates to at least one of image labelling, image tagging, image categorisation, video labelling, video tagging, video categorisation, handwriting recognition, identifying vulgar content, removing vulgar content, identifying illegal content, removing illegal content, duplicate checking, video transcription, and targeted photography of product placement, wherein the transmission occurs through a network configured for communication of the requester computing device and the worker computing device with the application server.