# US 2023/0067915 A1 · Diagnosing sources of noise in an evaluation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0067915 A1
- Document ID: US20230067915A1
- Status: Published application
- Kind: A1
- Application number: 17/880,534
- Filed: 2022-08-03
- Published: 2023-03-02
- Earliest claimed: 2020-10-01
- Assignees: CrowdSmart, Inc.
- Applicants: CrowdSmart, Inc.
- Inventors: Thomas Kehler; Markus Guehrs; Sonali Sinha
- Web page: https://patentgrep.com/patent/US20230067915A1

## CPC

- G06F 40/30: Semantic analysis
- G06F 16/245: Query processing
- G06F 16/355: Creation or modification of classes or clusters
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/24155: {Bayesian classification}
- G06F 18/295: {Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models}
- G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30)
- G06N 3/042: Knowledge-based neural networks; Logical representations of neural networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 5/02: Knowledge representation; Symbolic representation
- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G06N 5/04: Inference or reasoning models
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/00: Machine learning
- G06T 15/08: Volume rendering
- G06V 10/82: using neural networks
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/048: Activation functions
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Provided are processes of balancing between exploration and optimization with knowledge discovery processes applied to unstructured data with tight interrogation budgets. A process may include determining a relevance probability distribution of responses and scores as an explanatory diagnostic. A distribution curve may be determined based on a probabilistic graphical network and a result may be audited relative to the distribution curve to determine noise measurements. The distribution curve may be determined based on a distribution of posterior predictions of entities to score ranking entity bias and noisiness of ranking entity feedback.

## Claims

### Claim 1

A computer-implemented method comprising:
obtaining a probabilistic graphical network model based on a time series data set of feedback received from respective entities of a plurality of entities for one or more features corresponding to an evaluation;
obtaining observed data corresponding to a feature represented in the probabilistic graphical network model;
training a machine learning model based on a benchmark training data set corresponding to the feature;
determining, by the machine learning model, an observed score based on the observed data corresponding to the feature;
determining a distribution of posterior predictions based on the probabilistic graphical network model, the distribution based on a posterior prediction determined for each entity based on respective feedback including one or more scores encoded by the probabilistic graphical network model; and
determining, on a normalized scale, a distance between a peak of the distribution and the observed score, the distance being indicative of a bias of the entities for the feature.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
inferring a cause of variation in assessments of the evaluation among the plurality of entities based on relevance probability distributions of reasons given by the plurality of entities for the assessments.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
ranking the reasons according to the respective reasons causal contribution to the variation in assessments of the evaluation among the plurality of entities.

### Claim 4 (depends on 2)

The method of claim 2, further comprising:
generating a graphical user interface displaying relative contribution of each of a plurality of reasons to the variation in assessments of the evaluation among the plurality of entities.

### Claim 5 (depends on 4)

The method of claim 4, comprising sending instructions to display the graphical user interface to a remote client computer executing a web browser or native application in which the graphical user interface is to be displayed.

### Claim 6 (depends on 2)

The method of claim 2, comprising obtaining the time series data via an integration with a video conferencing application and displaying a result indicative of the cause of variation in assessments of the evaluation among the plurality of entities in the video conferencing application to the plurality of entities.

### Claim 7 (depends on 1)

The method of claim 1, wherein:
the plurality of entities comprises more than 10 entities;
obtaining observed data corresponding to a feature represented in the probabilistic graphical network model comprising obtaining observed data corresponding to more than 10 features represented in the probabilistic graphical network model; and
the probabilistic graphical network model comprises a Bayesian belief network.

### Claim 8 (depends on 1)

The method of claim 1, comprising:
obtaining the probabilistic graphical model by executing an iterative process by which the plurality of entities engage in structured deliberation about the evaluation.

### Claim 9 (depends on 1)

The method of claim 1, wherein the probabilistic graphical model models reasons of the entities for results of the evaluation.

### Claim 10 (depends on 1)

The method of claim 1, wherein the evaluation implements a linear decision model of a decision to be made collectively by the plurality of entities.

### Claim 11 (depends on 1)

The method of claim 1, comprising:
obtaining the feedback with a structured deliberation process integrated with team collaboration software; and
causing the team collaboration software to display the plot of the set of distributions.

### Claim 12 (depends on 1)

The method of claim 1, wherein the probabilistic graphical model comprises a Bayesian belief network.

### Claim 13 (depends on 1)

The method of claim 1, wherein the probabilistic graphical model comprises a probabilistic graphical network.

### Claim 14 (depends on 13)

The method of claim 13, wherein nodes of the probabilistic graphical network correspond to responses in the feedback from the plurality of entities and edges correspond to associations between the responses.

### Claim 15 (depends on 13)

The method of claim 13, wherein nodes of the probabilistic graphical network correspond to themes in the feedback from the plurality of entities and edges correspond to associations between the themes.

### Claim 16 (depends on 1)

The method of claim 1, comprising:
obtaining the feedback with steps for scaling A/B testing.

### Claim 17 (depends on 1)

The method of claim 1, comprising:
obtaining the feedback with steps for sampling a semantic space.

### Claim 18 (depends on 1)

The method of claim 1, comprising:
steps for determining measurements based on distributions determined based on the probabilistic graphical network model.

### Claim 19 (depends on 1)

The method of claim 1, wherein obtaining the probabilistic graphical model comprises steps for determining the probabilistic graphical model.

### Claim 20

A tangible, non-transitory, machine-readable medium storing instructions that, when executed by a computer system, effectuate operations comprising:
obtaining a probabilistic graphical network model based on a time series data set of feedback received from respective entities of a plurality of entities for one or more features corresponding to an evaluation;
obtaining observed data corresponding to a feature represented in the probabilistic graphical network model;
training a machine learning model based on a benchmark training data set corresponding to the feature;
determining, by the machine learning model, an observed score based on the observed data corresponding to the feature;
determining a distribution of posterior predictions based on the probabilistic graphical network model, the distribution based on a posterior prediction determined for each entity based on respective feedback including one or more scores encoded by the probabilistic graphical network model; and
determining, on a normalized scale, a distance between a peak of the distribution and the observed score, the distance being indicative of a bias of the entities for the feature.
