Automated data generation, post-job validation and agreement voting using automatic results generation and humans-in-the-loop, such as for tasks distributed by a crowdsourcing system

A system comprising:

• at least one hardware processor;
• and
• at least one non-transitory memory storing instructions, which, when executed by the at least one hardware processor, cause the system to:
• assign a first set of one or more instances of a first task to one or more crowd members, the first task constituting a particular type of activity, instances of the first task of the first set constituting different examples of the first task's type of activity, which are distinguished by having different instance data;
• collect results produced for at least a portion of the first set of instances of the first task by the crowd members to whom they were assigned in a first data generation step;
• use the collected results, together with the corresponding instance data, to train one or more machine learning models;
• assign a second set of one or more instances of the first task to the trained machine learning models;
• collect results produced for at least a portion of the second set of instances of the first task by the trained machine learning models to which they were assigned;
• and
• assign results produced for at least a portion of the first set of instances of the first task, or results produced for at least a portion of the second set of instances of the first task, or both for validation to the one or more crowd members.

A computer-readable storage medium, excluding transitory signals and carrying instructions, which, when executed by at least one data processor of a system, cause the system to:

• assign a first set of one or more instances of a first task to one or more crowd members, the first task constituting a particular type of activity, instances of the first task of the first set constituting different examples of the first task's type of activity, which are distinguished by having different instance data;
• collect results produced for at least a portion of the first set of instances of the first task by the crowd members to whom they were assigned in a first data generation step;
• use the collected results, together with the corresponding instance data, to train one or more machine learning models;
• assign a second set of one or more instances of the first task to the trained machine learning models;
• collect results produced for at least a portion of the second set of instances of the first task by the trained machine learning models to which they were assigned;
• and
• assign results produced for at least a portion of the first set of instances of the first task, or results produced for at least a portion of the second set of instances of the first task, or both for validation to the one or more crowd members.