# US 10,504,521 B1 · Training a dialog system using user feedback for answers to questions > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,504,521 B1 - Document ID: US10504521B1 - Status: Granted - Kind: B1 - Application number: 15/606,373 - Filed: 2017-05-26 - Granted: 2019-12-10 - Earliest claimed: 2012-06-01 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Gabriel Taubman; Andrew W. Hogue; John J. Lee - Web page: https://patentgrep.com/patent/US10504521B1 ## CPC - G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications - G06F 16/9535: Search customisation based on user profiles and personalisation - G06F 16/9536: Search customisation based on social or collaborative filtering - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G06F 16/3337: {Translation of the query language, e.g. Chinese to English} - H04M 3/493: Interactive information services, e.g. directory enquiries {; Arrangements therefor, e.g. interactive voice response [IVR] systems or voice portals} - H04M 3/4938: {comprising a voice browser which renders and interprets, e.g. VoiceXML} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08) ## Abstract Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a dialog system using user feedback. According to one implementation, a method includes receiving, by a dialog engine and from a computing device, a voice input; determining, by the dialog engine, that the received voice input is classified as feedback to an answer of a question; identifying, by the dialog engine, a predetermined feedback score associated with the feedback; and adjusting a confidence score associated with the question and the answer based on the predetermined feedback score. ## Claims ### Claim 1 A computer-implemented method comprising: receiving, at a dialog engine on a mobile device, a voice query from a user associated with the mobile device, the dialog engine configured to process spoken information from the user; in response to receiving the voice query: identifying, by the dialog engine, a set of multiple candidate answers for the voice query, each candidate answer having an associated confidence score that indicates an estimation of relevance of the candidate answer to the voice query; providing, by the dialog engine, a first answer to the voice query for output using a speaker of the mobile device, the first answer corresponding to the candidate answer associated with the highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query; receiving, at the dialog engine, feedback from the user indicating satisfaction or dissatisfaction with the provided first answer to the voice query; determining, by the dialog engine, whether the feedback was received within a predetermined amount of time of providing the first answer for output using the speaker of the mobile device; and when the feedback indicates dissatisfaction with the provided first answer and was received within the predetermined amount of time, providing, by the dialog engine, without receiving or detecting any additional activity from the user after receiving the feedback from the user, a second answer to the voice query for output using the speaker of the mobile device, the second answer corresponding to the candidate answer associated with the second highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein receiving feedback from the user comprises receiving a voice utterance detected by the mobile device that is uttered by the user. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising, when the feedback indicates dissatisfaction with the provided first answer and was received within the predetermined amount of time, updating, by the dialog engine, the confidence score associated with the candidate answer corresponding to the first answer. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, further comprising storing, by the dialog engine, multiple predefined queries that each correspond to a particular candidate answer. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, further comprising: determining, by the dialog engine, a level of similarity between the received voice query and one or more of the multiple predefined queries; and in response to determining that the level of similarity of the received voice query to a stored query exceeds a similarity threshold, obtaining, by the dialog engine, the particular predetermined answer that corresponds to the stored query, wherein the first answer is provided based on the particular predetermined answer that corresponds to the stored query. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein determining whether the feedback was received within the predetermined amount of time comprises determining that the user has performed one or more activities within the predetermined amount of time of providing the first answer for output using the speaker of the mobile device. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein the one or more activities include at least one of (i) calling a telephone number associated with the answer, (ii) sending an electronic message associated with the answer, or (iii) uttering one or more terms. ### Claim 8 A system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to execute a dialog engine on a mobile device and perform operations comprising: receiving, at the dialog engine, a voice query from a user associated with the mobile device, the dialog engine configured to process spoken information from the user; in response to receiving the voice query: identifying, a set of multiple candidate answers for the voice query, each candidate answer having an associated confidence score that indicates an estimation of relevance of the candidate answer to the voice query; providing a first answer to the voice query for output using a speaker of the mobile device, the first answer corresponding to the candidate answer associated with the highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query; receiving feedback from the user indicating satisfaction or dissatisfaction with the provided first answer to the voice query determining whether the feedback was received within a predetermined amount of time of providing the first answer for output using the speaker of the mobile device; and when the feedback indicates dissatisfaction with the provided first answer and was received within the predetermined amount of time, providing, without receiving or detecting any additional activity from the user after receiving the feedback from the user, a second answer to the voice query for output using the speaker of the mobile device, the second answer corresponding to the candidate answer associated with the second highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query. ### Claim 9 (depends on 8) The system of claim 8, wherein receiving feedback from the user comprises receiving a voice utterance detected by the mobile device that is uttered by the user. ### Claim 10 (depends on 8) The system of claim 8, wherein the operations further comprise, when the feedback indicates dissatisfaction with the provided first answer and was received within the predetermined amount of time, updating the confidence score associated with the candidate answer corresponding to the first answer. ### Claim 11 (depends on 8) The system of claim 8, wherein the operations further comprise: storing multiple predefined queries that each correspond to a particular predetermined answer; determining a level of similarity between the received voice query and one or more of the multiple predefined queries; and in response to determining that the level of similarity of the received voice query to a stored query exceeds a similarity threshold, obtaining the particular predetermined answer that corresponds to the stored query, wherein the audible output is based on the particular predetermined answer that corresponds to the stored query. ### Claim 12 (depends on 8) The system of claim 8, wherein determining whether the feedback was received within the predetermined amount of time comprises determining that the user has performed one or more activities within the predetermined amount of time of providing the first answer for output using the speaker of the mobile device. ### Claim 13 (depends on 12) The system of claim 12, wherein the one or more activities include at least one of (i) calling a telephone number associated with the answer, (ii) sending an electronic message associated with the answer, or (iii) uttering one or more terms. ### Claim 14 A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to execute a dialog engine on a mobile device and perform operations comprising: receiving, at the dialog engine, a voice query from a user associated with the mobile device, the dialog engine configured to process spoken information from the user; in response to receiving the voice query: identifying a set of multiple candidate answers for the voice query, each candidate answer having an associated confidence score that indicates an estimation of relevance of the candidate answer to the voice query; providing a first answer to the voice query for output using a speaker of the system, the first answer corresponding to the candidate answer associated with the highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query; receiving feedback from the user indicating satisfaction or dissatisfaction with the provided first answer to the voice query determining whether the feedback was received within a predetermined amount of time of providing the first answer for output using the speaker of the mobile device; and when the feedback indicates dissatisfaction with the provided first answer and was received within the predetermined amount of time, providing, without receiving or detecting any additional activity from the user after receiving the feedback from the user, a second answer to the voice query for output using the speaker of the mobile device, the second answer corresponding to the candidate answer associated with the second highest confidence score out of the confidence scores associated with the set of multiple candidate answers for the voice query. ### Claim 15 (depends on 14) The computer-readable medium of claim 14, wherein receiving feedback from the user comprises receiving a voice utterance detected by the mobile device that is uttered by the user. ### Claim 16 (depends on 14) The computer-readable medium of claim 14, wherein the operations further comprise, when the feedback indicates dissatisfaction with the provided first answer and was received within the predetermined amount of time, updating the confidence score associated with the candidate answer corresponding to the first answer. ### Claim 17 (depends on 14) The computer-readable medium of claim 14, wherein the operations further comprise storing multiple predefined queries that each correspond to a particular predetermined answer. ### Claim 18 (depends on 17) The computer-readable medium of claim 17, wherein the operations comprise: determining a level of similarity between the received voice query and one or more of the multiple predefined queries; and in response to determining that the level of similarity of the received voice query to a stored query exceeds a similarity threshold, obtaining the particular predetermined answer that corresponds to the stored query, wherein the first answer is provided based on the particular predetermined answer that corresponds to the stored query. ### Claim 19 (depends on 14) The computer-readable medium of claim 14, wherein determining whether the feedback was received within the predetermined amount of time comprises determining that the user has performed one or more activities within the predetermined amount of time of providing the first answer for output using the speaker of the mobile device. ### Claim 20 (depends on 19) The computer-readable medium of claim 19, wherein the one or more activities include at least one of (i) calling a telephone number associated with the answer, (ii) sending an electronic message associated with the answer, or (iii) uttering one or more terms.