Diagnosing sources of noise in an evaluation
AssigneeCrowdSmart, Inc.
InventorsThomas Kehler, Markus Guehrs, Sonali Sinha
Life of this patent
Estimating the term…
Abstract
Provided are processes of balancing between exploration and optimization with knowledge discovery processes applied to unstructured data with tight interrogation budgets. A process may include determining a relevance probability distribution of responses and scores as an explanatory diagnostic. A distribution curve may be determined based on a probabilistic graphical network and a result may be audited relative to the distribution curve to determine noise measurements. The distribution curve may be determined based on a distribution of posterior predictions of entities to score ranking entity bias and noisiness of ranking entity feedback.
Claims38
A computer-implemented method comprising:
- obtaining a probabilistic graphical network model based on a time series data set of feedback received from respective entities of a plurality of entities for one or more features corresponding to an evaluation;
- obtaining observed data corresponding to a feature represented in the probabilistic graphical network model;
- training a machine learning model based on a benchmark training data set corresponding to the feature;
- determining, by the machine learning model, an observed score based on the observed data corresponding to the feature;
- determining a distribution of posterior predictions based on the probabilistic graphical network model, the distribution based on a posterior prediction determined for each entity based on respective feedback including one or more scores encoded by the probabilistic graphical network model;
- and
- determining, on a normalized scale, a distance between a peak of the distribution and the observed score, the distance being indicative of a bias of the entities for the feature.
A tangible, non-transitory, machine-readable medium storing instructions that, when executed by a computer system, effectuate operations comprising:
- obtaining a probabilistic graphical network model based on a time series data set of feedback received from respective entities of a plurality of entities for one or more features corresponding to an evaluation;
- obtaining observed data corresponding to a feature represented in the probabilistic graphical network model;
- training a machine learning model based on a benchmark training data set corresponding to the feature;
- determining, by the machine learning model, an observed score based on the observed data corresponding to the feature;
- determining a distribution of posterior predictions based on the probabilistic graphical network model, the distribution based on a posterior prediction determined for each entity based on respective feedback including one or more scores encoded by the probabilistic graphical network model;
- and
- determining, on a normalized scale, a distance between a peak of the distribution and the observed score, the distance being indicative of a bias of the entities for the feature.
Dependencies are inferred from the claim text.