# US 11,030,239 B2 · Audio based entity-action pair based selection > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,030,239 B2 - Document ID: US11030239B2 - Status: Granted - Kind: B2 - Application number: 15/584,746 - Filed: 2017-05-02 - Published: 2018-08-09 - Granted: 2021-06-08 - Earliest claimed: 2013-05-31 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Alexander Collins; Ian James Leader; Yunkai Zhou; Gaurav Bhaya; Robert Stets - Web page: https://patentgrep.com/patent/US11030239B2 ## CPC - G06F 16/632: Query formulation - G06F 16/9032: Query formulation - G06F 16/951: Indexing; Web crawling techniques - G06F 16/953: Querying, e.g. by the use of web search engines - G06F 16/9532: Query formulation - G10L 15/08: Speech classification or search - G10L 25/54: for retrieval - G10L 2015/088: {Word spotting} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Routing packetized actions in a voice activated data packet based computer network environment is provided. A system can receive audio signals detected by a microphone of a device. The system can parse the audio signal to identify trigger keyword and request, and generate an action data structure. The action data structure can include digital components and entity-action pairs. ## Claims ### Claim 1 A system for transmission of packetized data in a voice activated computer network environment, the system comprising a data processing system having a processor and a memory to: receive, via an interface, an input audio signal detected by a microphone of a client computing device; identify a first search query within the input audio signal detected by the microphone of the client computing device; identify a first search entity based on the first search query; determine a first entity-action pair associated with the first search entity and an action of the first entity-action pair comprising a first action performed regarding the first search entity and determine a second entity-action pair associated with the first search entity and an action of the second entity-action pair comprising a second action, different than the first action, performed regarding the first search entity; select a first content item associated with the first entity-action pair based on a first quality score and a second content item associated with the second entity-action pair based on a second quality score; and transmit, to the client computing device, an output audio signal comprising the first content item based on a ranking of the first entity-action pair and the second entity-action pair. ### Claim 2 (depends on 1) The system of claim 1, further comprising the data processing system to: determine a plurality of entity-action pairs associated with the first search query; cluster the plurality of entity-action pairs into a first group of entity-action pairs and a second group of entity-action pairs based on each action of the plurality of entity-action pairs; rank, within the first group of entity-action pairs and the second group of entity-action pairs, the plurality of entity-action pairs; and select, based on the ranking of the plurality of entity-action pairs, the first entity-action pair from the first group of entity-action pairs and the second entity-action pair from the second group of entity-action pairs. ### Claim 3 (depends on 1) The system of claim 1, comprising the processor and the memory to: determine a number of times a search for a second search entity and a search for the first search entity originated from a same source; compare the number of times the search for the second search entity and the search for the first search entity originated from the same source to a predetermined threshold; and determine the first search entity is related to the second search entity responsive to the number of times the search for the second search entity and the search for the first search entity originated from the same source being above the predetermined threshold. ### Claim 4 (depends on 1) The system of claim 1, further comprising the data processing system to: receive, via the interface, a second input audio signal detected by the microphone of the client computing device; identify a second request comprising a second query term and not comprising a second trigger keyword within the second input audio signal detected by the microphone of the client computing device; determine a second search entity based on the second query term; rank a plurality of trigger keywords associated with the second query term; and identify the second trigger keyword from the ranked plurality of trigger keywords. ### Claim 5 (depends on 1) The system of claim 1, further comprising: a natural language processor component executed by the processor to receive, via an interface, data packets comprising the input audio signal; a natural language processor component to parse the input audio signal into the first search query; a direction action application programming interface to generate a first action data structure comprising the first entity-action pair. ### Claim 6 (depends on 5) The system of claim 5, further comprising the direction action application programming interface to: transmit the first action data structure to a third-party provider device to cause the third-party provider device to invoke a conversational application programming interface and establish a communication session between the third-party provider device and the client computing device. ### Claim 7 (depends on 5) The system of claim 5, further comprising the data processing system to: select, from a database, a template based on the first search query; identify a field in the template; receive a value from the client computing device corresponding to the field in the template; and populate the field in the template with the value received from the client computing device to generate the first entity-action pair. ### Claim 8 (depends on 1) The system of claim 1, further comprising: a content selector component to receive the first search query identified by a natural language processor and select, based on the first search query, the first content item via a real-time content selection process, the first content item provided by a content provider device. ### Claim 9 (depends on 1) The system of claim 1, comprising the processor and memory to: transmit the first entity-action pair to a third-party provider device to cause the third-party provider device to invoke a conversational application programming interface configured for the third-party provider device. ### Claim 10 (depends on 9) The system of claim 9, comprising the processor and memory to: route data packets of a communication session between the client computing device and the third-party provider device. ### Claim 11 A method of data transmission in a voice activated computer network environment, comprising: receiving, via an interface, an input audio signal detected by a microphone of a client computing device; identifying a first search query within the input audio signal detected by the microphone of the client computing device; determining a first search entity based on the first search query; determining a first entity-action pair associated with the first search entity and an action of the first entity-action pair comprising a first action performed regarding the first search entity; and determine a second entity-action pair associated with the first search entity and an action of the second entity-action pair comprising a second action, different than the first action, performed regarding the first search entity; selecting a first content item associated with the first entity-action pair based on a first quality score and a second content item associated with the second entity-action pair based on a second quality score; and transmitting, to the client computing device, an output audio signal comprising the first content item based on a ranking of the first entity-action pair and the second entity-action pair. ### Claim 12 (depends on 11) The method of claim 11, further comprising: determining a plurality of entity-action pairs associated with the first search query; clustering the plurality of entity-action pairs into groups of entity-action pairs based on each action of the plurality of entity-action pairs; ranking, within each of the groups of entity-action pairs, the plurality of entity-action pairs; and selecting, from each of the groups of entity-action pairs and based on the ranking of the plurality of entity-action pairs, a candidate entity-action pair. ### Claim 13 (depends on 11) The method of claim 11, further comprising: determining a number of times a search for a second search entity and a search for the first search entity originated from a same source; comparing the number of times the search for the second search entity and the search for the first search entity originated from the same source to a predetermined threshold; and determining the first search entity is related to the second search entity responsive to the number of times the search for the second search entity and the search for the first search entity originated from the same source being above the predetermined threshold. ### Claim 14 (depends on 11) The method of claim 11, further comprising: receiving, via the interface, a second input audio signal detected by the microphone of the client computing device; identifying a second request comprising a second query term and not comprising a second trigger keyword within the second input audio signal detected by the microphone of the client computing device; determining a second search entity based on the second query term; ranking a plurality of trigger keywords associated with the second query term; and identifying the second trigger keyword from the ranked plurality of trigger keywords. ### Claim 15 (depends on 11) The method of claim 11, further comprising: receiving, by a natural language processor component, data packets comprising the input audio signal; parsing, by a natural language processor component, the input audio signal into the first search query; and generating, by a direct action application programming interface, a first action data structure comprising the first entity-action pair. ### Claim 16 (depends on 15) The method of claim 15, further comprising: transmitting, by the direct action application programming interface, the first action data structure to a third party provider device to cause the third party provider device to invoke a conversational application programming interface and establish a communication session between the third party provider device and the client computing device. ### Claim 17 (depends on 15) The method of claim 15, further comprising: selecting, from a database, a template based on the first search query; identifying a field in the template; receiving a value from the client computing device corresponding to the field in the template; and populating the field in the template with the value received from the client computing device to generate the first entity-action pair. ### Claim 18 (depends on 15) The method of claim 15, comprising: receiving, by a content selector component, the first search query identified by a natural language processor; and selecting, based on the first search query, the first content item via a real-time content selection process, the first content item provided by a content provider device. ### Claim 19 (depends on 11) The method of claim 11, further comprising: transmitting the first entity-action pair to a third-party provider device to cause a third party provider device to invoke a conversational application programming interface configured for the third party provider device. ### Claim 20 (depends on 19) The method of claim 19, further comprising: routing data packets of a communication session between the client computing device and the third party provider device.