Probabilistic multigraph modeling for improving the quality of crowdsourced affective data

A system for improving quality of crowdsourced affective data based on agreement relationship between a plurality of annotators, comprising:

• a processor;
• a memory, the memory containing programming instructions thereon that, when executed, cause the processor to complete the steps including:
• receiving a collection of stimuli, all the stimuli previously given affective labels by the plurality of annotators;
• constructing a multigraph agreement model including a pair-wise status of agreement between the affective labels given by different ones of the plurality of annotators, the multigraph agreement model constructed as a probabilistic model;
• learning the probabilistic model computationally using the crowdsourced affective data to generate a learned model;
• repeatedly:
• assessing a confidence score of each stimuli based on the learned model, the confidence score being whether the stimuli has a confidence of evoking consensus emotion responses;
• identifying a reliability of each of the plurality of annotators based on the learned model;
• and adjusting the crowdsourced affective data by:
• recalculating the affective labels of each stimuli based on the identified reliability of each of the plurality of annotators, thereby improving the quality of the crowdsourced affective data, wherein learning one of the confidence score and reliability is recursively enhanced on what is learned of an other of the confidence score and reliability.