# US 2021/0334306 A1 · Coordination of overlapping processing of audio queries > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0334306 A1 - Document ID: US20210334306A1 - Status: Published application - Kind: A1 - Application number: 16/341,763 - Filed: 2019-04-08 - Published: 2021-10-28 - Earliest claimed: 2018-05-03 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Bo Wang; Smita Rai; Max Ohlendorf; Subbaiah Venkata; Chad Yoshikawa; Abhinav Taneja; Amit Agarwal; Chris Ramsdale; Chris Turkstra - Web page: https://patentgrep.com/patent/US20210334306A1 ## CPC - G06F 16/2471: {Distributed queries} - G06F 16/3329: Natural language query formulation - G06F 16/632: Query formulation - G06F 16/634: {Query by example, e.g. query by humming} - G06F 16/638: Presentation of query results - G06F 21/6218: {to a system of files or objects, e.g. local or distributed file system or database} - G06F 40/35: Discourse or dialogue representation - H04L 67/02: based on web technology, e.g. hypertext transfer protocol [HTTP] - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract Coordinating processing of audio queries is provided. A system receives a query. The system provides the query to a first digital assistant component and a second digital assistant component for processing. The system receives a first response to the query from the first digital assistant component, and a second response to the query from the second digital assistant component. The first digital assistant component can be authorized to access a database the second digital assistant component is prohibited from accessing. The system determines, based on a ranking decision function, to select the second response to the query from the second digital assistant component. The system provides, responsive to the selection, the second response from the second digital assistant to a computing device. ## Claims ### Claim 1 A system to coordinate overlapping processing of audio queries, comprising: a data processing system comprising at least one processor and memory to: receive, via an interface, one or more data packets corresponding to an audio signal detected by a sensor of a computing device; generate a query based on the audio signal received via the one or more data packets; transmit the query to a plurality of digital assistant components; receive a first response to the query from a first digital assistant component of the plurality of digital assistant components; receive a second response to the query from a second digital assistant component of the plurality of digital assistant components, wherein the first digital assistant component is authorized to access a database the second digital assistant component is prohibited from accessing; determine, based on a ranking decision function, to select the second response from the second digital assistant component instead of the first response from the first digital assistant component; and provide, responsive to the selection, the second response from the second digital assistant component to the computing device. ### Claim 2 (depends on 1) The system of claim 1, comprising: the data processing system to select the second response from the second digital assistant component based on a pre-configured preference established for the second digital assistant component. ### Claim 3 (depends on 1) The system of claim 1, comprising: the data processing system to select the second response from the second digital assistant component based on a confidence score for the second response from the second digital assistant component. ### Claim 4 (depends on 1) The system of claim 1, comprising the data processing system to: determine the first digital assistant component is a preferred digital assistant; determine that the first response from the first digital assistant component is flagged as a fallback response; override, responsive to the determination that the first response from the first digital assistant component is flagged as the fallback response, the first digital assistant component; select the second response from the second digital assistant component; and maintain the first digital assistant component as the preferred digital assistant. ### Claim 5 (depends on 1) The system of claim 1, comprising the data processing system to: determine a signal for the query; and select the ranking decision function from a plurality of ranking decision functions based on the signal. ### Claim 6 (depends on 1) The system of claim 1, comprising the data processing system to: determine a signal for the query; and select responses from the second digital assistant component based on the signal. ### Claim 7 (depends on 1) The system of claim 1, comprising: the data processing system to determine, based on the ranking decision function, to select the second response based on: the second digital assistant component accessing data stored in memory of the computing device. ### Claim 8 (depends on 1) The system of claim 1, wherein the first response comprises one of calendar information or media content. ### Claim 9 (depends on 1) The system of claim 1, wherein the first digital assistant component executes independently of the second digital assistant component to generate the first response. ### Claim 10 (depends on 1) The system of claim 1, comprising: the data processing system to transmit the query to each of the plurality of digital assistants via an independent branch to cause the plurality of digital assistants to process the query in an overlapping manner. ### Claim 11 (depends on 1) The system of claim 1, wherein the first digital assistant component is established by a different administrator from the second digital assistant component. ### Claim 12 (depends on 1) The system of claim 1, comprising the data processing system to: receive the one or more data packets from the computing device via a streaming speech server configured to convert audio to text. ### Claim 13 (depends on 1) The system of claim 1, comprising the data processing system to: receive the second response from the second digital assistant component prior to the first response from the first digital assistant component; determine that the second digital assistant component is not a preferred digital assistant component; determine that the first digital assistant is the preferred digital assistant; wait, responsive to the determination that the second digital assistant component is not a preferred digital assistant and the first digital assistant is the preferred digital assistant, for the first response from the first digital assistant component; upon receipt of the first response from the first digital assistant component, determine that the first response is flagged as a fallback response; and select the second response based on the determination that the second digital assistant component is not the preferred digital assistant and the first response is flagged as the fallback response. ### Claim 14 A method of coordinating overlapping processing of audio queries, comprising: receiving, by a data processing system comprising a processor and memory, one or more data packets corresponding to an audio signal detected by a sensor of a computing device; generating, by the data processing system, a query based on the audio signal received via the one or more data packets; transmitting, by the data processing system, the query to a plurality of digital assistant components; receiving, by the data processing system, a first response to the query from a first digital assistant component of the plurality of digital assistant components; receiving, by the data processing system, a second response to the query from a second digital assistant component of the plurality of digital assistant components, wherein the first digital assistant component is authorized to access a database the second digital assistant component is prohibited from accessing; determining, by the data processing system, based on a ranking decision function, to select the second response from the second digital assistant component instead of the first response from the first digital assistant component; and providing, by the data processing system responsive to the selection, the second response from the second digital assistant component to the computing device. ### Claim 15 (depends on 14) The method of claim 14, comprising: selecting the second response from the second digital assistant component based on a confidence score for the second response from the second digital assistant component. ### Claim 16 (depends on 14) The method of claim 14, comprising: determining the first digital assistant component is a preferred digital assistant; determining that the first response from the first digital assistant component is flagged as a fallback response; overriding, responsive to the determination that the first response from the first digital assistant component is flagged as the fallback response, the first digital assistant component; selecting the second response from the second digital assistant component; and maintaining the first digital assistant component as the preferred digital assistant. ### Claim 17 (depends on 14) The method of claim 14, comprising: determining, based on the ranking decision function, to select the first response based on the first digital assistant component accessing data stored in memory of the computing device. ### Claim 18 (depends on 14) The method of claim 14, comprising: transmitting the query to each of the plurality of digital assistants via an independent branch to cause the plurality of digital assistants to process the query in an overlapping manner. ### Claim 19 (depends on 14) The method of claim 14, comprising: receiving the second response from the second digital assistant component prior to the first response from the first digital assistant component; determining that the second digital assistant component is not a preferred digital assistant component; determining that the first digital assistant is the preferred digital assistant; waiting, responsive to the determination that the second digital assistant component is not a preferred digital assistant and the first digital assistant is the preferred digital assistant, for the first response from the first digital assistant component; upon receipt of the first response from the first digital assistant component, determining that the first response is flagged as a fallback response; and selecting the second response based on the determination that the second digital assistant component is not the preferred digital assistant and the first response is flagged as the fallback response. ### Claim 20 (canceled) ### Claim 21 (depends on 1) The system of claim 1, comprising: the data processing system to determine, based on the ranking decision function, to select the second response based on the first response being an audio-only response.