# US 10,372,737 B2 · Automatic identification of retraining data in a classifier-based dialogue system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,372,737 B2 - Document ID: US10372737B2 - Status: Granted - Kind: B2 - Application number: 15/814,473 - Filed: 2017-11-16 - Published: 2019-05-16 - Granted: 2019-08-06 - Earliest claimed: 2017-11-16 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Allen Ginsberg; Edward G. Katz; Alexander C. Tonetti - Web page: https://patentgrep.com/patent/US10372737B2 ## CPC - G06F 16/3329: Natural language query formulation - G06F 40/35: Discourse or dialogue representation - G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning} - G10L 15/183: using context dependencies, e.g. language models - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 2015/0631: {Creating reference templates; Clustering} - G10L 2015/0635: {updating or merging of old and new templates; Mean values; Weighting} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract According to one embodiment, a method, computer system, and computer program product for retraining a classifier-based automatic dialog system with recorded user interactions is provided. The present invention may include receiving recorded interactions, where the interactions are between a user and an automatic dialog system; determining, based on the recorded interactions, whether to pair a given input with one or more classes; pairing inputs with one or more classes; assessing the reliability of the paired inputs and classes; determining whether the reliable paired inputs and classes can be consistently mapped; and merging all consistently mapped reliable pairs with an initial training set. ## Claims ### Claim 1 A processor-implemented method for identifying data for retraining a classifier-based automatic dialogue system with recorded user interactions, the method comprising: receiving one or more recorded interactions, wherein the one or more recorded interactions comprise one or more inputs by a user and one or more responses by the automatic dialogue system; determining, based on the one or more recorded interactions, whether to pair one or more inputs with one or more classes; pairing, based on the one or more recorded interactions, the one or more inputs with one or more classes; assessing, based on the one or more recorded interactions, a reliability of the one or more paired inputs and classes; determining whether the one or more reliable paired inputs and classes can be consistently mapped; and merging all consistently-mapped, reliable, paired inputs and classes with an initial training set. ### Claim 2 (depends on 1) The method of claim 1, wherein the one or more recorded interactions further comprise: one or more classes, ranked according to a confidence level, for each of the one or more inputs; and for one or more classes where the confidence level is below a threshold, one or more system responses asking the user for confirmation that a class with a highest confidence level is appropriate. ### Claim 3 (depends on 1) The method of claim 1, wherein determining whether to pair the one or more inputs with the one or more classes further comprises: analyzing the one or more recorded interactions to identify whether the user indicated a class to be appropriate or inappropriate. ### Claim 4 (depends on 1) The method of claim 1, wherein assessing the reliability of the one or more paired inputs and classes further comprises: comparing a number of times a class has been mapped to an input against a threshold. ### Claim 5 (depends on 1) The method of claim 1, wherein a given paired input and class is consistently mapped if the one or more inputs are mapped to the same one or more classes over one or more user interactions, and wherein a number of outlying classes fall below a threshold value. ### Claim 6 (depends on 1) The method of claim 1, further comprising: for one or more paired inputs and classes which are not consistently mapped, determining whether a consistent submapping can be generated. ### Claim 7 (depends on 6) The method of claim 6, wherein determining whether a consistent submapping uses one or more methods selected from a group consisting of: comparing a percentage that any of one or more classes paired with an input are selected against a threshold, and sending a plurality of inconsistently mapped paired inputs and classes to one or more users for manual determination. ### Claim 8 A computer system for identifying data for retraining a classifier-based automatic dialogue system with recorded user interactions, the computer system comprising: one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage medium, and program instructions stored on at least one of the one or more tangible storage medium for execution by at least one of the one or more processors via at least one of the one or more memories, wherein the computer system is capable of performing a method comprising: receiving one or more recorded interactions, wherein the one or more recorded interactions comprise one or more inputs by a user and one or more responses by the automatic dialogue system; determining, based on the one or more recorded interactions, whether to pair one or more inputs with one or more classes; pairing, based on the one or more recorded interactions, the one or more inputs with one or more classes; assessing, based on the one or more recorded interactions, a reliability of the one or more paired inputs and classes; determining whether the one or more reliable paired inputs and classes can be consistently mapped; and merging all consistently-mapped, reliable, paired inputs and classes with an initial training set. ### Claim 9 (depends on 8) The computer system of claim 8, wherein the one or more recorded interactions further comprise: one or more classes, ranked according to a confidence level, for each of the one or more inputs; and for one or more classes where the confidence level is below a threshold, one or more system responses asking the user for confirmation that a class with a highest confidence level is appropriate. ### Claim 10 (depends on 8) The computer system of claim 8, wherein determining whether to pair the one or more inputs with the one or more classes further comprises: analyzing the one or more recorded interactions to identify whether the user indicated a class to be appropriate or inappropriate. ### Claim 11 (depends on 8) The computer system of claim 8, wherein assessing the reliability of the one or more paired inputs and classes further comprises: includes comparing a number of times a class has been mapped to an input against a threshold. ### Claim 12 (depends on 8) The computer system of claim 8, wherein a given paired input and class is consistently mapped if the one or more inputs are mapped to the same one or more classes over one or more user interactions, and wherein a number of outlying classes fall below a threshold value. ### Claim 13 (depends on 8) The computer system of claim 8, further comprising: for one or more paired inputs and classes which are not consistently mapped, determining whether a consistent submapping can be generated. ### Claim 14 (depends on 8) The computer system of claim 8, wherein determining whether a consistent submapping uses one or more methods selected from a group consisting of: comparing a percentage that any of one or more classes paired with an input are selected against a threshold, and sending a plurality of inconsistently mapped paired inputs and classes to one or more users for manual determination. ### Claim 15 A computer program product for identifying data for retraining a classifier-based automatic dialogue system with recorded user interactions, the computer program product comprising: one or more computer-readable tangible storage medium and program instructions stored on at least one of the one or more tangible storage medium, the program instructions executable by a processor to cause the processor to perform a method comprising: receiving one or more recorded interactions, wherein the one or more recorded interactions comprise one or more inputs by a user and one or more responses by the automatic dialogue system; determining, based on the one or more recorded interactions, whether to pair one or more inputs with one or more classes; pairing, based on the one or more recorded interactions, the one or more inputs with one or more classes; assessing, based on the one or more recorded interactions, a reliability of the one or more paired inputs and classes; determining whether the one or more reliable paired inputs and classes can be consistently mapped; and merging all consistently-mapped, reliable, paired inputs and classes with an initial training set. ### Claim 16 (depends on 15) The computer program product of claim 15, wherein the one or more recorded interactions further comprise: one or more classes, ranked according to a confidence level, for each of the one or more inputs; and for one or more classes where the confidence level is below a threshold, one or more system responses asking the user for confirmation that a class with a highest confidence level is appropriate. ### Claim 17 (depends on 15) The computer program product of claim 15, wherein determining whether to pair the one or more inputs with the one or more classes further comprises: analyzing the one or more recorded interactions to identify whether the user indicated a class to be appropriate or inappropriate. ### Claim 18 (depends on 15) The computer program product of claim 15, wherein assessing the reliability of the one or more paired inputs and classes further comprises: comparing a number of times a class has been mapped to an input against a threshold. ### Claim 19 (depends on 15) The computer program product of claim 15, wherein a given paired input and class is consistently mapped if the one or more inputs are mapped to the same one or more classes over one or more user interactions, and wherein a number of outlying classes fall below a threshold value. ### Claim 20 (depends on 15) The computer program product of claim 15, further comprising: for one or more paired inputs and classes which are not consistently mapped, determining whether a consistent submapping can be generated.