# US 12,112,754 B2 · Automated assistant for facilitating communications through dissimilar messaging features of different applications > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,754 B2 - Document ID: US12112754B2 - Status: Granted - Kind: B2 - Application number: 18/514,730 - Filed: 2023-11-20 - Published: 2024-03-14 - Granted: 2024-10-08 - Earliest claimed: 2020-12-02 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Victor Carbune; Matthew Sharifi - Web page: https://patentgrep.com/patent/US12112754B2 ## CPC - G06F 40/30: Semantic analysis - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 40/295: Named entity recognition - G06N 20/00: Machine learning - G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning} - H04L 51/224: providing notification on incoming messages, e.g. pushed notifications of received messages - H04L 51/56: Unified messaging, e.g. interactions between e-mail, instant messaging or converged IP messaging [CPM] - G10L 2015/223: {Execution procedure of a spoken command} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract Implementations relate to an automated assistant that can respond to communications received via a third party application and/or other third party communication modality. The automated assistant can determine that the user is participating in multiple different conversations via multiple different third party communication services. In some implementations, conversations can be processed to identify particular features of the conversations. When the automated assistant is invoked to provide input to a conversation, the automated assistant can compare the input to the identified conversation features in order to select the particular conversation that is most relevant to the input. In this way, the automated assistant can assist with any of multiple disparate conversations that are each occurring via a different third party application. ## Claims ### Claim 1 A method implemented by one or more processors, the method comprising: receiving a spoken utterance of a user, wherein the spoken utterance is detected via a computing device and is directed to an automated assistant that is accessible via the computing device; determining, based on the spoken utterance, that the user is requesting that the automated assistant communicate a message to another user, wherein the spoken utterance identifies content to be incorporated into the message; processing, based on the spoken utterance, application content from one or more applications accessible via the computing device, wherein in processing the application content, existing application data corresponding to the spoken utterance is identified; comparing, based on the spoken utterance, the existing application data to natural language content of the spoken utterance, determining, based on comparing the existing application data and the natural language content, that the existing application data corresponds to the content to be incorporated into the message; generating, responsive to determining that the existing application data corresponds to the content to be incorporated into the message, a command for a particular application to render the message; causing, by the automated assistant and based on generating the command for the particular application to render the message, the command to be provided to the particular application, wherein the input is provided to the particular application in furtherance of communicating the message to the other user. ### Claim 2 (depends on 1) The method of claim 1, wherein the existing application data is based on prior interactions between the user and the one or more applications. ### Claim 3 (depends on 2) The method of claim 2, further comprising: generating, based on the natural language content, an utterance embedding; and prior to receiving the spoken utterance: processing the existing application data, and generating, based on processing the existing application data, one or more existing application data embeddings. ### Claim 4 (depends on 3) The method of claim 3, wherein comparing the existing application data to the natural language content of the spoken utterance comprises: comparing the existing application data embeddings to the utterance embedding. ### Claim 5 (depends on 4) The method of claim 4, wherein at least one of the interaction embeddings corresponds to the existing conversation data. ### Claim 6 (depends on 1) The method of claim 1, further comprising: prior to receiving the spoken utterance, determining that the particular application has provided a notification to the user via the computing device and that the user has previously provided an input to the particular application; and generating annotation data based on the notification, wherein the annotation data corresponds to the input previously provided to the particular application by the user. ### Claim 7 (depends on 6) The method of claim 6, wherein determining that the existing application data corresponds to the content to be incorporated into the message is further based on the annotation data. ### Claim 8 (depends on 6) The method of claim 6, wherein determining that the existing application data corresponds to the content to be incorporated into the message further comprises: generating semantic data based on the natural language content of the spoken utterance; comparing the semantic data to the annotation data; determining a correspondence between the semantic data and the annotation data; and determining, based on the correspondence, that the existing application data corresponds to the content to be incorporated into the message. ### Claim 9 A system comprising: memory storing instructions; one or more processors operable to execute the instructions to: receive a spoken utterance of a user, wherein the spoken utterance is detected via one or more microphones and is directed to an automated assistant; determine, based on the spoken utterance, that the user is requesting that the automated assistant communicate a message to another user, wherein the spoken utterance identifies content to be incorporated into the message; process, based on the spoken utterance, application content from one or more applications, wherein in processing the application content, existing application data corresponding to the spoken utterance is identified; compare, based on the spoken utterance, the existing application data to natural language content of the spoken utterance, determine, based on comparing the existing application data and the natural language content, that the existing application data corresponds to the content to be incorporated into the message; generate, responsive to determining that the existing application data corresponds to the content to be incorporated into the message, a command for a particular application to render the message; cause, by the automated assistant and based on generating the command for the particular application to render the message, the command to be provided to the particular application, wherein the input is provided to the particular application in furtherance of communicating the message to the other user. ### Claim 10 (depends on 9) The system of claim 9, wherein the existing application data is based on prior interactions between the user and the one or more applications. ### Claim 11 (depends on 10) The system of claim 10, wherein one or more of the processors are further operable to execute the instructions to: generate, based on the natural language content, an utterance embedding; and prior to receiving the spoken utterance: process the existing application data, and generate, based on processing the existing application data, one or more existing application data embeddings. ### Claim 12 (depends on 11) The system of claim 11, wherein in comparing the existing application data to the natural language content of the spoken utterance one or more of the processors are to: compare the existing application data embeddings to the utterance embedding. ### Claim 13 (depends on 12) The system of claim 12, wherein at least one of the interaction embeddings corresponds to the existing conversation data. ### Claim 14 (depends on 9) The system of claim 9, one or more of the processors are further operable to execute the instructions to: prior to receiving the spoken utterance, determine that the particular application has provided a notification to the user and that the user has previously provided an input to the particular application; and generate annotation data based on the notification, wherein the annotation data corresponds to the input previously provided to the particular application by the user. ### Claim 15 (depends on 14) The system of claim 14, wherein in determining that the existing application data corresponds to the content to be incorporated into the message, one or more of the processors are to determine that the existing application data corresponds to the content to be incorporated into the message further based on the annotation data. ### Claim 16 (depends on 14) The system of claim 14, wherein in determining that the existing application data corresponds to the content to be incorporated into the message one or more of the processors are to: generate semantic data based on the natural language content of the spoken utterance; compare the semantic data to the annotation data; determine a correspondence between the semantic data and the annotation data; and determine, based on the correspondence, that the existing application data corresponds to the content to be incorporated into the message.