Techniques for increasing the accuracy of subjective quality experiments

One or more non-transitory computer readable media including instructions that, when executed by one or more processors, cause the one or more processors to mitigate scoring inaccuracies in subjective quality experiments, by performing the steps of:

• performing, via a propagation solver, a plurality of iterative operations to update a first plurality of parameters of a model that includes a plurality of individual scores assigned by a plurality of participants during a first experiment round to generate a first subjective factor set comprising a different participant inconsistency for each participant in the plurality of participants, wherein performing the plurality of iterative operations includes generating a maximum likelihood estimation formulation of the model and executing the propagation solver to automatically compute a solution for the maximum likelihood estimation formulation to generate the first subjective factor set comprising initiating an update rule generator to generate a set of update rules for updating the first plurality of parameters, and iteratively initiating a propagation engine to iteratively update the first plurality of parameters based on the set of update rules and a refresh rate until a stop threshold is satisfied to generate an optimized plurality of parameters for the model that comprises the solution for the maximum likelihood estimation formulation, wherein the first subjective factor set optimized plurality of parameters corresponds to the first subjective factor set;
• computing a ranking of the plurality of participants with respect to each other based on the first subjective factor set;
• and
• generating a participant evaluation report that includes information indicating a portion of the plurality of participants that, based on the ranking of the plurality of participants with respect to each other, meet a participant inconsistency criterion for exclusion from a second experiment round during which the plurality of participants less the portion of the plurality of participants assign one or more additional individual scores, wherein the participant inconsistency criterion specifies a predefined number of participants included in the plurality of participants that are associated with highest participant consistencies.

A system comprising:

• one or more memories storing instructions;
• and
• one or more processors coupled to the one or more memories that, when executing the instructions, perform the steps of:
• performing, via a propagation solver, a plurality of iterative operations to update a first plurality of parameters of a model that includes a plurality of individual scores assigned by a plurality of participants during a first experiment round to generate a first subjective factor set comprising a different participant inconsistency for each participant in the plurality of participants, wherein performing the plurality of iterative operations includes generating a maximum likelihood estimation formulation of the model and executing the propagation solver to automatically compute a solution for the maximum likelihood estimation formulation to generate the first subjective factor set comprising initiating an update rule generator to generate a set of update rules for updating the first plurality of parameters, and iteratively initiating a propagation engine to iteratively update the first plurality of parameters based on the set of update rules and a refresh rate until a stop threshold is satisfied to generate an optimized plurality of parameters for the model that comprises the solution for the maximum likelihood estimation formulation, wherein the first subjective factor set optimized plurality of parameters corresponds to the first subjective factor set;
• computing a ranking of the plurality of participants with respect to each other based on the first subjective factor set;
• and
• generating a participant evaluation report that includes information indicating a portion of the plurality of participants that, based on the ranking of the plurality of participants with respect to each other, meet a participant inconsistency criterion for exclusion from a second experiment round during which the plurality of participants less the portion of the plurality of participants assign one or more additional individual scores, wherein the participant inconsistency criterion specifies a predefined number of participants included in the plurality of participants that are associated with highest participant consistencies.