# US 12,764,302 B2 · Techniques for increasing the accuracy of subjective quality experiments > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,764,302 B2 - Document ID: US12764302B2 - Status: Granted - Kind: B2 - Application number: 17/840,544 - Filed: 2022-06-14 - Published: 2022-12-01 - Granted: 2026-10-13 - Earliest claimed: 2020-07-31 - Assignees: Netflix, Inc. - Applicants: Netflix, Inc. - Inventors: Zhi Li - Web page: https://patentgrep.com/patent/US12764302B2 ## CPC - H04N 19/149: by estimating the code amount by means of a model, e.g. mathematical model or statistical model - G06Q 30/0203: Market surveys; Market polls - H04L 65/80: Responding to QoS - H04N 19/154: Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion (use of rate-distortion criteria H04N19/147) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract In various embodiments, a data optimization application mitigates scoring inaccuracies in subjective quality experiments. In operation, the data optimization application generates a model that includes a first set of individual scores and a first set of parameters. The first set of parameters includes a first subjective score set and a first set of subjective factor sets. The data optimization application performs one or more optimization operations on the first set of parameters to generate a second set of parameters. The second set of parameters includes a second subjective score set and a second set of subjective factor sets, wherein the second subjective score set compensates for at least a first subjective factor set included in the second set of subjective factor sets. The data optimization application also computes a participant evaluation report based on at least a second subjective factor sets included in the second set of subjective factor sets ## Claims ### Claim 1 A computer-implemented method for mitigating scoring inaccuracies in subjective quality experiments, the method comprising: performing, via a propagation solver, a plurality of iterative operations to update a first plurality of parameters of a model that includes a plurality of individual scores assigned by a plurality of participants during a first experiment round to generate a first subjective factor set comprising a different participant inconsistency for each participant in the plurality of participants, wherein performing the plurality of iterative operations includes generating a maximum likelihood estimation formulation of the model and executing the propagation solver to automatically compute a solution for the maximum likelihood estimation formulation to generate the first subjective factor set comprising initiating an update rule generator to generate a set of update rules for updating the first plurality of parameters, and iteratively initiating a propagation engine to iteratively update the first plurality of parameters based on the set of update rules and a refresh rate until a stop threshold is satisfied to generate an optimized plurality of parameters for the model that comprises the solution for the maximum likelihood estimation formulation, wherein the first subjective factor set optimized plurality of parameters corresponds to the first subjective factor set; computing a ranking of the plurality of participants with respect to each other based on the first subjective factor set; and generating a participant evaluation report that includes information indicating a portion of the plurality of participants that, based on the ranking of the plurality of participants with respect to each other, meet a participant inconsistency criterion for exclusion from a second experiment round during which the plurality of participants less the portion of the plurality of participants assign one or more additional individual scores, wherein the participant inconsistency criterion specifies a predefined number of participants included in the plurality of participants that are associated with highest participant consistencies. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the first plurality of parameters includes a first subjective score set and a first plurality of subjective factor sets. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein performing the plurality of iterative operations comprises performing one or more optimization operations on the first plurality of parameters of the model to generate a second plurality of parameters that includes a second subjective score set and a second plurality of subjective factor sets, wherein the second subjective score set compensates for at least a second subjective factor set included in the second plurality of subjective factor sets, and wherein the second plurality of subjective factor sets includes the first subjective factor set. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, wherein the first subjective factor set comprises a participant inconsistency set, and the second subjective factor set comprises a participant bias set. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein generating the participant evaluation report comprises generating a ranked participant set based on the ranking, wherein the participant evaluation report includes the ranked participant set. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the participant evaluation report comprises a rejection recommendation that specifies at least one participant associated with the plurality of individual scores. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the maximum likelihood estimation formulation of the model is generated based on a log maximum likelihood function. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein a first individual score included in the plurality of individual scores comprises an assessment of visual quality for a reconstructed video clip derived from an encoded video clip. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the refresh rate specifies a magnitude of the updates to the first plurality of parameters and a rate at which the propagation engine approaches the solution for the maximum likelihood estimation formulation. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the stop threshold specifies an acceptable accuracy for the solution for the maximum likelihood estimation formulation. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein the propagation solver further initiates an initial value generator to compute initial values for the first plurality of parameters and initiates the update rule generator to simplify the maximum likelihood estimation formulation. ### Claim 12 One or more non-transitory computer readable media including instructions that, when executed by one or more processors, cause the one or more processors to mitigate scoring inaccuracies in subjective quality experiments, by performing the steps of: performing, via a propagation solver, a plurality of iterative operations to update a first plurality of parameters of a model that includes a plurality of individual scores assigned by a plurality of participants during a first experiment round to generate a first subjective factor set comprising a different participant inconsistency for each participant in the plurality of participants, wherein performing the plurality of iterative operations includes generating a maximum likelihood estimation formulation of the model and executing the propagation solver to automatically compute a solution for the maximum likelihood estimation formulation to generate the first subjective factor set comprising initiating an update rule generator to generate a set of update rules for updating the first plurality of parameters, and iteratively initiating a propagation engine to iteratively update the first plurality of parameters based on the set of update rules and a refresh rate until a stop threshold is satisfied to generate an optimized plurality of parameters for the model that comprises the solution for the maximum likelihood estimation formulation, wherein the first subjective factor set optimized plurality of parameters corresponds to the first subjective factor set; computing a ranking of the plurality of participants with respect to each other based on the first subjective factor set; and generating a participant evaluation report that includes information indicating a portion of the plurality of participants that, based on the ranking of the plurality of participants with respect to each other, meet a participant inconsistency criterion for exclusion from a second experiment round during which the plurality of participants less the portion of the plurality of participants assign one or more additional individual scores, wherein the participant inconsistency criterion specifies a predefined number of participants included in the plurality of participants that are associated with highest participant consistencies. ### Claim 13 (depends on 12) The one or more non-transitory computer readable media of claim 12, wherein the first plurality of parameters includes a first subjective score set and a first plurality of subjective factor sets. ### Claim 14 (depends on 13) The one or more non-transitory computer readable media of claim 13, wherein performing the plurality of iterative operations comprises performing one or more optimization operations on the first plurality of parameters of the model to generate a second plurality of parameters that includes a second subjective score set and a second plurality of subjective factor sets, wherein the second plurality of subjective factor sets includes the first subjective factor set, and wherein the second subjective score set compensates for at least a second subjective factor set included in the second plurality of subjective factor sets. ### Claim 15 (depends on 14) The one or more non-transitory computer readable media of claim 14, wherein the first subjective factor set comprises a participant inconsistency set, and the second subjective factor set comprises a participant bias set. ### Claim 16 (depends on 14) The one or more non-transitory computer readable media of claim 14, wherein the second subjective score set includes a perceptual video quality score that estimates visual quality of a reconstructed video clip as perceived by a hypothetical consistent and unbiased subject. ### Claim 17 (depends on 12) The one or more non-transitory computer readable media of claim 12, the steps further comprising generating a ranked participant set based on the ranking, wherein the participant evaluation report includes the ranked participant set. ### Claim 18 (depends on 12) The one or more non-transitory computer readable media of claim 12, wherein the participant evaluation report comprises a rejection recommendation, and further comprising computing the rejection recommendation by: determining at least one participant associated with the plurality of individual scores based on at least the first subjective factor set and the participant inconsistency criterion; and generating the rejection recommendation specifying the at least one participant. ### Claim 19 A system comprising: one or more memories storing instructions; and one or more processors coupled to the one or more memories that, when executing the instructions, perform the steps of: performing, via a propagation solver, a plurality of iterative operations to update a first plurality of parameters of a model that includes a plurality of individual scores assigned by a plurality of participants during a first experiment round to generate a first subjective factor set comprising a different participant inconsistency for each participant in the plurality of participants, wherein performing the plurality of iterative operations includes generating a maximum likelihood estimation formulation of the model and executing the propagation solver to automatically compute a solution for the maximum likelihood estimation formulation to generate the first subjective factor set comprising initiating an update rule generator to generate a set of update rules for updating the first plurality of parameters, and iteratively initiating a propagation engine to iteratively update the first plurality of parameters based on the set of update rules and a refresh rate until a stop threshold is satisfied to generate an optimized plurality of parameters for the model that comprises the solution for the maximum likelihood estimation formulation, wherein the first subjective factor set optimized plurality of parameters corresponds to the first subjective factor set; computing a ranking of the plurality of participants with respect to each other based on the first subjective factor set; and generating a participant evaluation report that includes information indicating a portion of the plurality of participants that, based on the ranking of the plurality of participants with respect to each other, meet a participant inconsistency criterion for exclusion from a second experiment round during which the plurality of participants less the portion of the plurality of participants assign one or more additional individual scores, wherein the participant inconsistency criterion specifies a predefined number of participants included in the plurality of participants that are associated with highest participant consistencies.