# US 2020/0401914 A1 · Systems and methods for automatically detecting and repairing slot errors in machine learning training data for a machine learning-based dialogue system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401914 A1 - Document ID: US20200401914A1 - Status: Published application - Kind: A1 - Application number: 16/895,147 - Filed: 2020-06-08 - Published: 2020-12-24 - Earliest claimed: 2019-06-21 - Assignees: Clinc, Inc. - Applicants: Clinc, Inc. - Inventors: Stefan Larson; Anish Mahendran; Parker Hill; Jonathan K. Kummerfeld; Michael A. Laurenzano; Lingjia Tang; Jason Mars - Web page: https://patentgrep.com/patent/US20200401914A1 ## CPC - G06N 5/027: {Frames} - G06N 5/04: Inference or reasoning models - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/09: Supervised learning - G06N 20/00: Machine learning - G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06N 20/20: Ensemble learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Systems and methods for automatically detecting annotation discrepancies in annotated training data samples and repairing the annotated training data samples for a machine learning-based automated dialogue system include evaluating a corpus of a plurality of distinct training data samples; identifying one or more of a slot span defect and a slot label defect of a target annotated slot span of a target training data sample of the corpus based on the evaluation; and automatically correcting one or more annotations of the target annotated slot span based on the identified one or more of the slot span defect and the slot label defect. ## Claims ### Claim 1 A method for automatically detecting annotation discrepancies in annotated training data samples and repairing the annotated training data samples for a machine learning-based automated dialogue system, the method comprising: evaluating a corpus of a plurality of distinct training data samples; identifying one or more of a slot span defect and a slot label defect of a target annotated slot span of a target training data sample of the corpus based on the evaluation; and automatically correcting one or more annotations of the target annotated slot span based on the identified one or more of the slot span defect and the slot label defect. ### Claim 2 (depends on 1) The method according to claim 1, wherein automatically correcting the one or more annotations of the target annotated slot span includes one of: automatically expanding the target annotated slot span to include one or more surrounding tokens based on identifying the slot span defect; and automatically contracting the target annotated slot span to exclude one or more tokens within the annotated slot span based on detecting the slot span defect. ### Claim 3 (depends on 1) The method according to claim 1, wherein automatically correcting the one or more annotations of the target annotated slot span includes one or more of: deleting one or more erroneous slot labels from the target annotated slot span based on identifying the slot label defect, and annotating to the target annotated slot span a proper slot label based on slot labels applied to a majority of like slot spans within the corpus. ### Claim 4 (depends on 1) The method according to claim 1, wherein the slot span defect relates to an error or an inconsistency in a setting of a length of a given annotated slot span of a given training data sample, wherein the length of the given annotated slot span is set such that the given annotated slot span erroneously includes one or more extraneous tokens or erroneously excludes one or more pertinent tokens of the given training data sample. ### Claim 5 (depends on 1) The method according to claim 1, wherein the slot label defect relates to an error or an inconsistency in an application of one or more distinct slot classification labels to a given annotated slot span of a given training data sample. ### Claim 6 (depends on 1) The method according to claim 1, further comprising: constructing a slot span format evaluator for the target annotated slot span based on span format attributes of one or more of the plurality of distinct training data samples of the corpus, wherein constructing the slot span format evaluator includes: defining a left- and right n-gram set for a slot type of the target annotated slot span, wherein the slot type relates to one of a plurality of distinct slot classification labels. ### Claim 7 (depends on 6) The method according to claim 6, wherein: a given left n-gram set of the left- and right n-gram set comprises a set of token sequences that excludes a last token in an n-gram, a given right n-gram set of the left- and right n-gram sets comprises a set of token sequences that excludes the first token in the n-gram, and the n-gram is a sequence of tokens of a given training data sample within the corpus of annotated training samples. ### Claim 8 (depends on 6) The method according to claim 6, wherein: evaluating the corpus of the plurality of distinct training data samples includes iterating the slot format evaluator over each of the plurality of distinct training data samples within the corpus, wherein iterating the slot format evaluator includes: identifying whether a segment of a given annotated slot span of a given training data sample is positioned adjacent to an un-labeled span of the given training data sample having a member token within the left- and right n-gram set for a slot type of the given annotated training data sample; and if the segment of a given annotated slot span is adjacent to the unlabeled span that includes the member token, identifying the member token as a candidate for inclusion into the given annotated slot span. ### Claim 9 (depends on 8) The method according to claim 8, wherein evaluating the corpus of the plurality of distinct training data samples includes: implementing a voting stage that informs a candidate exclusion policy or a candidate inclusion policy for the member token, wherein the voting stage includes: identifying candidate-slot pairs from within the corpus, wherein each candidate-slot pair includes a pairing of the member token with an adjacent token as found throughout distinct training data samples within the corpus, wherein the voting stage informs the candidate inclusion policy if a majority of candidate-slot pairs occur labeled as part of a single slot span thereby adding the member token to the given annotated slot span. ### Claim 10 (depends on 9) The method according to claim 9, further comprising: designating the given annotated slot span as an archetypal slot span for the slot type of the target annotated slot span, and identifying whether the target annotated slot span includes the slot span defect is based on a comparison of the target annotated slot span to the archetypal slot span. ### Claim 11 (depends on 6) The method according to claim 6, wherein: evaluating the corpus of the plurality of distinct training data samples includes iterating the slot format evaluator over each of the plurality of distinct training data samples within the corpus, wherein iterating the slot format evaluator includes: identifying whether a given annotated slot span is positioned adjacent to an un-labeled span not having a member token within the left- and right n-gram set for a slot type of the given annotated slot span; and if the given annotated slot span is adjacent to the un-labeled span that does not include the member token, identifying a token within the given annotated slot span that is adjacent to the un-labeled slot span as a candidate for exclusion from the given annotated slot span. ### Claim 12 (depends on 11) The method according to claim 11, wherein evaluating the corpus of the plurality of distinct training data samples includes: implementing a voting stage that informs a candidate exclusion policy or a candidate inclusion policy for the member token, wherein the voting stage includes: identifying candidate-slot pairs from within the corpus, wherein each candidate-slot pair includes a pairing of the member token with an adjacent token as found throughout the plurality of distinct training data samples within the corpus, wherein the voting stage informs the candidate exclusion policy if a majority of candidate-slot pairs do not occur labeled as part of a single annotated slot span thereby excluding the member token from the given annotated slot span. ### Claim 13 (depends on 12) The method according to claim 12, further comprising: designating the given annotated slot span as an archetypal slot span for the slot type of the target annotated slot span, and identifying whether the target annotated slot span includes the slot span defect is based on a comparison of the target annotated slot span to the archetypal slot span. ### Claim 14 (depends on 1) The method according to claim 1, further comprising: implementing a label variation evaluator, wherein the label variation evaluator includes a variation n-gram that identifies whether there is the slot label defect in the given annotated training data sample, wherein implementing the label variation evaluator includes: (1) setting the given annotated slot span as a slot nucleus; (2) setting a fixed radius, k, around the nucleus, where k is a predetermined number of tokens; and (3) identify all variation n-grams for the given training sample based on the slot nucleus and fixed radius, k, wherein each identified variation n-gram includes an n-gram that includes a combination of the slot nucleus and one or more tokens surrounding the slot nucleus within a given training data sample. ### Claim 15 (depends on 14) The method according to claim 14, further comprising: implementing a voting stage that identifies whether a standard labeling convention for the variation n-gram exists within the corpus of annotated training sample data, wherein the standard labeling convention exists if a majority of annotated training data samples having the variation n-gram within the corpus include a distinct slot classification label, wherein if the standard labeling convention exists, the label variation evaluator determines whether the slot label annotated to the given training data sample is defective. ### Claim 16 (depends on 14) The method according to claim 14, wherein if the slot label annotated to the given training data sample is defective, automatically repairing an annotation of the given annotated slot span to remove the slot label and ascribe a new slot label defined by the standard labeling convention to the given annotated slot span. ### Claim 17 (depends on 1) The method according to claim 1, wherein the slot span defect includes one or more identified slot span defect types including one or more of: an omission inconsistency; and an addition inconsistency. ### Claim 18 (depends on 17) The method according to claim 17, wherein automatically correcting the one or more annotations of the target annotated slot span is further based on the one or more identified slot span defect types. ### Claim 19 (depends on 1) The method according to claim 1, wherein the slot label defect includes one or more identified slot label defect types including one or more of: a wrong label inconsistency; a swapped label inconsistency; and a chop and join inconsistency. ### Claim 20 (depends on 19) The method according to claim 19, wherein automatically correcting the one or more annotations of the target annotated slot span is further based on the one or more identified slot label defect types.