# US 12,112,760 B2 · Managing dialog data providers > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,760 B2 - Document ID: US12112760B2 - Status: Granted - Kind: B2 - Application number: 18/222,325 - Filed: 2023-07-14 - Published: 2023-11-09 - Granted: 2024-10-08 - Earliest claimed: 2015-07-31 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: David Kliger Elson; David P. Whipp; Shir Judith Yehoshua - Web page: https://patentgrep.com/patent/US12112760B2 ## CPC - G10L 17/22: Interactive procedures; Man-machine interfaces - G06F 16/3329: Natural language query formulation - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for managing dialogs. In one aspect, a method includes receiving a request associated with a task from a user device; submitting the request to each of a plurality of distinct data providers; receiving a plurality of suggested dialog responses from two or more of the data providers; scoring the one or more suggested dialog responses based on one or more scoring factors; determining a particular dialog response to provide to the user based on the scoring; and providing the determined dialog response to the user device. ## Claims ### Claim 1 A method implemented by one or more processors, the method comprising: receiving, from a user device, a request comprising a first voice input; submitting the request to each of a plurality of distinct data providers, wherein each data provider is associated with a distinct data model, and wherein the plurality of distinct data providers includes (i) a task provider that directs a dialog toward information used to complete a task and (ii) a search provider; in response to the first voice input, receiving at least one suggested response to the first voice input from at least one of the plurality of distinct data providers; determining, from the at least one suggested response to the first voice input, a first dialog including one or more responses to the first voice input to provide to the user device to complete the first dialog; receiving, from the user device, a second voice input in response to providing the one or more responses to the first voice input to the user device, and submitting the second voice input to each of the plurality of distinct data providers; in response to the second voice input, receiving suggested responses to the second voice input from two or more of the plurality of distinct data providers, the suggested responses to the second voice input including a suggested response to the second voice input from the search provider and a suggested response to the second voice input from the task provider, the suggested response to the second voice input from the task provider being based on the suggested response to the second voice input from the search provider; for each suggested response to the second voice input, determining a respective score associated with the suggested response; selecting, from the suggested responses to the second voice input, a response to the second voice input based on the respective scores associated with the suggested responses to the second voice input; and providing the response to the second voice input to the user device. ### Claim 2 (depends on 1) The method of claim 1, wherein determining the respective score associated with the suggested response is based on a user personalization model. ### Claim 3 (depends on 1) The method of claim 1, wherein determining the respective score associated with the suggested response is based on a respective confidence score that is included with the suggested response that is received from one of the data providers. ### Claim 4 (depends on 1) The method of claim 1, comprising: updating a state of the first dialog in response to the one or more responses to the first voice input; and providing the updated state to the plurality of data providers as context for analyzing the second voice input. ### Claim 5 (depends on 1) The method of claim 1, wherein: the at least one suggested response to the first voice input includes a suggested response to the first voice input from the task provider; and the one or more responses to the first voice input include the suggested response to the first voice input from the task provider. ### Claim 6 (depends on 1) The method of claim 1, wherein the suggested responses to the second voice input further include an additional suggested response to the second voice input from the task provider, the additional suggested response to the second voice input from the task provider not being based on the suggested response to the second voice input from the search provider. ### Claim 7 (depends on 1) The method of claim 1, wherein: the plurality of distinct data providers further includes an additional data provider; the suggested responses to the second voice input further include a suggested response to the second voice input from the additional data provider and an additional suggested response to the second voice input from the task provider, the additional suggested response to the second voice input from the task provider being based on the suggested response to the second voice input from the additional data provider. ### Claim 8 (depends on 7) The method of claim 7, wherein a score associated with the suggested response to the second voice input from the additional data provider does not satisfy a threshold, further comprising, in response to the score associated with the suggested response to the second voice input from the additional data provider not satisfying the threshold: avoiding basing the response to the second voice input on the suggested response to the second voice input from the additional data provider; and avoiding basing the response to the second voice input on the additional suggested response to the second voice input from the task provider. ### Claim 9 A computer program product comprising one or more non-transitory computer-readable storage media having program instructions collectively stored on the one or more computer-readable storage media, the program instructions executable to: receive, from a user device, a request comprising a first voice input; submit the request to each of a plurality of distinct data providers, wherein each data provider is associated with a distinct data model, and wherein the plurality of distinct data providers includes (i) a task provider that directs a dialog toward information used to complete a task and (ii) a search provider; in response to the first voice input, receive at least one suggested response to the first voice input from at least one of the plurality of distinct data providers; determine, from the at least one suggested response to the first voice input, a first dialog including one or more responses to the first voice input to provide to the user device to complete the first dialog; receive, from the user device, a second voice input in response to providing the one or more responses to the first voice input to the user device, and submitting the second voice input to each of the plurality of distinct data providers; in response to the second voice input, receive suggested responses to the second voice input from two or more of the plurality of distinct data providers, the suggested responses to the second voice input including a suggested response to the second voice input from the search provider and a suggested response to the second voice input from the task provider, the suggested response to the second voice input from the task provider being based on the suggested response to the second voice input from the search provider; for each suggested response to the second voice input, determine a respective score associated with the suggested response; select, from the suggested responses to the second voice input, a response to the second voice input based on the respective scores associated with the suggested responses to the second voice input; and provide the response to the second voice input to the user device. ### Claim 10 (depends on 9) The computer program product according to claim 9, wherein determining the respective score associated with the suggested response is based on a user personalization model. ### Claim 11 (depends on 9) The computer program product according to claim 9, wherein determining the respective score associated with the suggested response is based on a respective confidence score that is included with the suggested response that is received from one of the data providers. ### Claim 12 (depends on 9) The computer program product according to claim 9, wherein the program instructions are further executable to: update a state of the first dialog in response to the one or more responses to the first voice input; and provide the updated state to the plurality of data providers as context for analyzing the second voice input. ### Claim 13 (depends on 9) The computer program product according to claim 9, wherein: the at least one suggested response to the first voice input includes a suggested response to the first voice input from the task provider; and the one or more responses to the first voice input include the suggested response to the first voice input from the task provider. ### Claim 14 (depends on 9) The computer program product according to claim 9, wherein the suggested responses to the second voice input further include an additional suggested response to the second voice input from the task provider, the additional suggested response to the second voice input from the task provider not being based on the suggested response to the second voice input from the search provider. ### Claim 15 A system comprising: a processor, a computer-readable memory, one or more computer-readable storage media, and program instructions collectively stored on the one or more computer-readable storage media, the program instructions executable to: receive, from a user device, a request comprising a first voice input; submit the request to each of a plurality of distinct data providers, wherein each data provider is associated with a distinct data model, and wherein the plurality of distinct data providers includes (i) a task provider that directs a dialog toward information used to complete a task and (ii) a search provider; in response to the first voice input, receive at least one suggested response to the first voice input from at least one of the plurality of distinct data providers; determine, from the at least one suggested response to the first voice input, a first dialog including one or more responses to the first voice input to provide to the user device to complete the first dialog; receive, from the user device, a second voice input in response to providing the one or more responses to the first voice input to the user device, and submitting the second voice input to each of the plurality of distinct data providers; in response to the second voice input, receive suggested responses to the second voice input from two or more of the plurality of distinct data providers, the suggested responses to the second voice input including a suggested response to the second voice input from the search provider and a suggested response to the second voice input from the task provider, the suggested response to the second voice input from the task provider being based on the suggested response to the second voice input from the search provider; for each suggested response to the second voice input, determine a respective score associated with the suggested response; select, from the suggested responses to the second voice input, a response to the second voice input based on the respective scores associated with the suggested responses to the second voice input; and provide the response to the second voice input to the user device. ### Claim 16 (depends on 15) The system according to claim 15, wherein determining the respective score associated with the suggested response is based on a user personalization model. ### Claim 17 (depends on 15) The system according to claim 15, wherein determining the respective score associated with the suggested response is based on a respective confidence score that is included with the suggested response that is received from one of the data providers. ### Claim 18 (depends on 15) The system according to claim 15, wherein the program instructions are further executable to: update a state of the first dialog in response to the one or more responses to the first voice input; and provide the updated state to the plurality of data providers as context for analyzing the second voice input. ### Claim 19 (depends on 15) The system according to claim 15, wherein: the at least one suggested response to the first voice input includes a suggested response to the first voice input from the task provider; and the one or more responses to the first voice input include the suggested response to the first voice input from the task provider. ### Claim 20 (depends on 15) The system according to claim 15, wherein the suggested responses to the second voice input further include an additional suggested response to the second voice input from the task provider, the additional suggested response to the second voice input from the task provider not being based on the suggested response to the second voice input from the search provider.