# US 2022/0277221 A1 · System and method for improving machine learning training data quality > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0277221 A1 - Document ID: US20220277221A1 - Status: Published application - Kind: A1 - Application number: 17/405,875 - Filed: 2021-08-18 - Published: 2022-09-01 - Earliest claimed: 2021-02-26 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Tapas Kanungo; Nehal Bengre; Stephen Walsh; Pingjie Tang - Web page: https://patentgrep.com/patent/US20220277221A1 ## CPC - G06N 20/00: Machine learning - G06N 20/20: Ensemble learning - G06N 3/09: Supervised learning - G06N 5/04: Inference or reasoning models - G06N 3/045: Combinations of networks - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A method includes generating, using at least one processor of an electronic device, a plurality of expert labels for a sample using a plurality of machine learned classifiers. The method also includes determining, using the at least one processor, an expert consensus label among the plurality of expert labels. The method further includes comparing, using the at least one processor, the expert consensus label to a ground truth label associated with the sample in response to determining that a consensus is found among the plurality of machine learned classifiers. The method also includes identifying, using the at least one processor, the ground truth label as a clean label in response to determining that the expert consensus label and the ground truth label match. In addition, the method includes identifying, using the at least one processor, the ground truth label for reassessment in response to determining that the expert consensus label and the ground truth label do not match. ## Claims ### Claim 1 A method comprising: generating, using at least one processor of an electronic device, a plurality of expert labels for a sample using a plurality of machine learned classifiers; determining, using the at least one processor, an expert consensus label among the plurality of expert labels; comparing, using the at least one processor, the expert consensus label to a ground truth label associated with the sample in response to determining that a consensus is found among the plurality of machine learned classifiers; identifying, using the at least one processor, the ground truth label as a clean label in response to determining that the expert consensus label and the ground truth label match; and identifying, using the at least one processor, the ground truth label for reassessment in response to determining that the expert consensus label and the ground truth label do not match. ### Claim 2 (depends on 1) The method of claim 1, further comprising: identifying, among multiple guidelines corresponding to the ground truth label, at least one guideline that needs to be revised based on a degree of mismatch between the expert consensus label and the ground truth label. ### Claim 3 (depends on 2) The method of claim 2, further comprising: determining whether to reassess the sample using the at least one guideline after the at least one guideline is revised. ### Claim 4 (depends on 2) The method of claim 2, wherein the ground truth label is generated by a grader using the multiple guidelines corresponding to the ground truth label. ### Claim 5 (depends on 1) The method of claim 1, further comprising: determining that a lack of consensus is found among the plurality of machine learned classifiers; and marking the sample for reassessment in response to determining that the lack of consensus is found among the plurality of machine learned classifiers. ### Claim 6 (depends on 1) The method of claim 1, wherein the machine learned classifiers are trained using multi-fold cross validation. ### Claim 7 (depends on 1) The method of claim 1, wherein the machine learned classifiers include different types of classifiers selected to reduce bias in label generation. ### Claim 8 (depends on 1) The method of claim 1, wherein the consensus is based on a largest number of matches among the plurality of expert labels. ### Claim 9 (depends on 1) The method of claim 1, wherein: the sample is one of a plurality of samples; and each of the samples is associated with a verbal utterance. ### Claim 10 An electronic device comprising: at least one memory configured to store instructions; and at least one processing device configured when executing the instructions to: generate a plurality of expert labels for a sample using a plurality of machine learned classifiers; determine an expert consensus label among the plurality of expert labels; compare the expert consensus label to a ground truth label associated with the sample in response to determining that a consensus is found among the plurality of machine learned classifiers; identify the ground truth label as a clean label in response to determining that the expert consensus label and the ground truth label match; and identify the ground truth label for reassessment in response to determining that the expert consensus label and the ground truth label do not match. ### Claim 11 (depends on 10) The electronic device of claim 10, wherein the at least one processing device is further configured to identify, among multiple guidelines corresponding to the ground truth label, at least one guideline that needs to be revised based on a degree of mismatch between the expert consensus label and the ground truth label. ### Claim 12 (depends on 11) The electronic device of claim 11, wherein the at least one processing device is further configured to determine whether to reassess the sample using the at least one guideline after the at least one guideline is revised. ### Claim 13 (depends on 11) The electronic device of claim 11, wherein the ground truth label is generated by a grader using the multiple guidelines corresponding to the ground truth label. ### Claim 14 (depends on 10) The electronic device of claim 10, wherein the at least one processing device is further configured to: determine that a lack of consensus is found among the plurality of machine learned classifiers; and mark the sample for reassessment in response to determining that the lack of consensus is found among the plurality of machine learned classifiers. ### Claim 15 (depends on 10) The electronic device of claim 10, wherein the machine learned classifiers are trained using multi-fold cross validation. ### Claim 16 (depends on 10) The electronic device of claim 10, wherein the machine learned classifiers include different types of classifiers selected to reduce bias in label generation. ### Claim 17 (depends on 10) The electronic device of claim 10, wherein the consensus is based on a largest number of matches among the plurality of expert labels. ### Claim 18 (depends on 10) The electronic device of claim 10, wherein: the sample is one of a plurality of samples; and each of the samples is associated with a verbal utterance. ### Claim 19 A non-transitory machine-readable medium containing instructions that when executed cause at least one processor of an electronic device to: generate a plurality of expert labels for a sample using a plurality of machine learned classifiers; determine an expert consensus label among the plurality of expert labels; compare the expert consensus label to a ground truth label associated with the sample in response to determining that a consensus is found among the plurality of machine learned classifiers; identify the ground truth label as a clean label in response to determining that the expert consensus label and the ground truth label match; and identify the ground truth label for reassessment in response to determining that the expert consensus label and the ground truth label do not match. ### Claim 20 (depends on 19) The non-transitory machine-readable medium of claim 19, further comprising instructions that when executed cause at least one processor to identify, among multiple guidelines corresponding to the ground truth label, at least one guideline that needs to be revised based on a degree of mismatch between the expert consensus label and the ground truth label.