# US 2022/0035643 A1 · Initializing a conversation with an automated agent via selectable graphical element > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0035643 A1 - Document ID: US20220035643A1 - Status: Published application - Kind: A1 - Application number: 17/504,049 - Filed: 2021-10-18 - Published: 2022-02-03 - Earliest claimed: 2017-04-25 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Vikram Aggarwal; Dina Elhaddad - Web page: https://patentgrep.com/patent/US20220035643A1 ## CPC - G06F 9/453: {Help systems} - G06F 3/04812: Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G10L 15/1822: {Parsing for meaning understanding} - G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Methods, apparatus, systems, and computer-readable media are provided for using selectable elements to invoke an automated assistant at a computing device. While operating the computing device, a user may not be aware that the automated assistant can be invoked according to certain invocation phrases. In order to inform the user of the functionality of the automated assistant, the user can be presented with selectable elements that can initialize the automated assistant when selected. Furthermore, a selectable element can provide an invocation phrase in textual form so that the user is aware of their ability to invoke the automated assistant by speaking the invocation phrase. The selectable element can be presented at different devices associated with the user, and the automated assistant can be initialized at a device that is separate from the device where the selectable element is presented. ## Claims ### Claim 1 A computer-implemented method, comprising: receiving, by a server device, an indication of a user selection of a selectable element rendered at a graphical user interface of a first computing device, wherein the selectable element is rendered at the graphical user interface of the first computing device while an application is executing at the first computing device, and wherein the selectable element is rendered at the graphical user interface with a command phrase that includes natural language content, which, when provided as a spoken input by a user to an automated assistant interface, prior to the selectable element being rendered at the graphical user interface of the first computing device, transitions an automated assistant into an operating state in which the automated assistant causes an operation associated with the application to be performed; and in response to receiving the indication of the user selection of the selectable element: identifying multiple computing devices associated with an account of the user associated with the first computing device, wherein the multiple computing devices include at least the first computing device and one or more additional computing devices; selecting a particular computing device of the one or more additional computing devices for initializing the automated assistant, from among the multiple computing devices, based on the operation associated with the application that is to be performed; and causing, in response to the selection of the selectable element and based on selecting the particular computing device, the automated assistant to: initialize at the automated assistant interface of the particular computing device, perform the operation associated with the application, transition a state in which the automated assistant monitors, at the automated assistant interface of the particular computing device, for one or more additional spoken utterances of the user at the automated assistant interface of the particular computing device, and provide one or more additional selectable elements at the automated assistant interface of the particular computing device, the one or more additional selectable elements being selectable by the user to cause the automated assistant at the automated assistant interface of the particular computing device to perform one or more additional operations associated with the application. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising: providing, to the server device, credentials of the user; and receiving, from the server device and based on the credentials of the user, a directory of computing devices associated with the user, wherein the directory identifies the one or more additional computing devices. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising: causing the first computing device or the particular computing device to audibly render the phrase. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the command phrase has not been previously submitted by user via any spoken input to the automated assistant. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, further comprising: providing, to the server device, data that identifies application content previously displayed at the graphical user interface while the application is executing and prior to rendering the command phrase with the selectable element, wherein at least a portion of the command phrase describes the previously displayed application content. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the server device causes the selectable element to be rendered based on data associated with an account profile of the user. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein the operation associated with the application is performed using a particular value for a parameter, and wherein the server device determines the particular value for the parameter based on a frequency that the user has previously used the application to perform the operation using the particular value for the parameter. ### Claim 8 A system comprising: memory and one or more processors, the memory storing instructions that, when executed by the one or more processors, cause the one or more processors to: receive, by a server device, an indication of a user selection of a selectable element rendered at a graphical user interface of a first computing device, wherein the selectable element is rendered at the graphical user interface of the first computing device while an application is executing at the first computing device, and wherein the selectable element is rendered at the graphical user interface with a command phrase that includes natural language content, which, when provided as a spoken input by a user to an automated assistant interface, prior to the selectable element being rendered at the graphical user interface of the first computing device, transitions an automated assistant into an operating state in which the automated assistant causes an operation associated with the application to be performed; and in response to receiving the indication of the user selection of the selectable element: identify multiple computing devices associated with an account of the user associated with the first computing device, wherein the multiple computing devices include at least the first computing device and one or more additional computing devices; select a particular computing device of the one or more additional computing devices for initializing the automated assistant, from among the multiple computing devices, based on the operation associated with the application that is to be performed; and cause, in response to the selection of the selectable element and based on selecting the particular computing device, the automated assistant to: initialize at the automated assistant interface of the particular computing device, perform the operation associated with the application, transition a state in which the automated assistant monitors, at the automated assistant interface of the particular computing device, for one or more additional spoken utterances of the user at the automated assistant interface of the particular computing device, and provide one or more additional selectable elements at the automated assistant interface of the particular computing device, the one or more additional selectable elements being selectable by the user to cause the automated assistant at the automated assistant interface of the particular computing device to perform one or more additional operations associated with the application. ### Claim 9 (depends on 8) The system of claim 8, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to: provide, to the server device, credentials of the user; and receive, from the server device and based on the credentials of the user, a directory of computing devices associated with the user, wherein the directory identifies the one or more additional computing devices. ### Claim 10 (depends on 8) The system of claim 8, further comprising: causing the first computing device or the particular computing device to audibly render the phrase. ### Claim 11 (depends on 8) The system of claim 8, wherein the command phrase has not been previously submitted by user via any spoken input to the automated assistant. ### Claim 12 (depends on 11) The computer-implemented method of claim 11, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to: provide, to the server device, data that identifies application content previously displayed at the graphical user interface while the application is executing and prior to rendering the command phrase with the selectable element, wherein at least a portion of the command phrase describes the previously displayed application content. ### Claim 13 (depends on 8) The system of claim 8, wherein the server device causes the selectable element to be rendered based on data associated with an account profile of the user. ### Claim 14 (depends on 13) The system of claim 13, wherein the operation associated with the application is performed using a particular value for a parameter, and wherein the server device determines the particular value for the parameter based on a frequency that the user has previously used the application to perform the operation using the particular value for the parameter. ### Claim 15 At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations: receive, by a server device, an indication of a user selection of a selectable element rendered at a graphical user interface of a first computing device, wherein the selectable element is rendered at the graphical user interface of the first computing device while an application is executing at the first computing device, and wherein the selectable element is rendered at the graphical user interface with a command phrase that includes natural language content, which, when provided as a spoken input by a user to an automated assistant interface, prior to the selectable element being rendered at the graphical user interface of the first computing device, transitions an automated assistant into an operating state in which the automated assistant causes an operation associated with the application to be performed; and in response to receiving the indication of the user selection of the selectable element: identify multiple computing devices associated with an account of the user associated with the first computing device, wherein the multiple computing devices include at least the first computing device and one or more additional computing devices; select a particular computing device of the one or more additional computing devices for initializing the automated assistant, from among the multiple computing devices, based on the operation associated with the application that is to be performed; and cause, in response to the selection of the selectable element and based on selecting the particular computing device, the automated assistant to: initialize at the automated assistant interface of the particular computing device, perform the operation associated with the application, transition a state in which the automated assistant monitors, at the automated assistant interface of the particular computing device, for one or more additional spoken utterances of the user at the automated assistant interface of the particular computing device, and provide one or more additional selectable elements at the automated assistant interface of the particular computing device, the one or more additional selectable elements being selectable by the user to cause the automated assistant at the automated assistant interface of the particular computing device to perform one or more additional operations associated with the application. ### Claim 16 (depends on 15) The at least one non-transitory computer-readable medium of claim 15, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to: provide, to the server device, credentials of the user; and receive, from the server device and based on the credentials of the user, a directory of computing devices associated with the user, wherein the directory identifies the one or more additional computing devices. ### Claim 17 (depends on 15) The at least one non-transitory computer-readable medium of claim 15, further comprising: causing the first computing device or the particular computing device to audibly render the phrase. ### Claim 18 (depends on 15) The at least one non-transitory computer-readable medium of claim 15, wherein the command phrase has not been previously submitted by user via any spoken input to the automated assistant. ### Claim 19 (depends on 18) The at least one non-transitory computer-readable medium of claim 18, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to: provide, to the server device, data that identifies application content previously displayed at the graphical user interface while the application is executing and prior to rendering the command phrase with the selectable element, wherein at least a portion of the command phrase describes the previously displayed application content. ### Claim 20 (depends on 15) The at least one non-transitory computer-readable medium of claim 15, wherein the server device causes the selectable element to be rendered based on data associated with an account profile of the user.