# US 12,112,097 B2 · Facilitating discovery of verbal commands using multimodal interfaces > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,097 B2 - Document ID: US12112097B2 - Status: Granted - Kind: B2 - Application number: 18/152,055 - Filed: 2023-01-09 - Published: 2023-05-25 - Granted: 2024-10-08 - Earliest claimed: 2019-03-15 - Assignees: Adobe Inc. - Applicants: Adobe Inc. - Inventors: Lubomira Dontcheva; Arjun Srinivasan; Seth John Walker; Eytan Adar - Web page: https://patentgrep.com/patent/US12112097B2 ## CPC - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 3/0483: Interaction with page-structured environments, e.g. book metaphor - G06F 3/0487: using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser - G06F 9/451: Execution arrangements for user interfaces - G06F 9/453: {Help systems} - G06F 16/686: {using information manually generated, e.g. tags, keywords, comments, title or artist information, time, location or usage information, user ratings} - G06F 16/90324: {using system suggestions} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer - G10L 2015/223: {Execution procedure of a spoken command} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A framework for generating and presenting verbal command suggestions to facilitate discoverability of commands capable of being understood and support users exploring available commands. A target associated with a direct-manipulation input is received from a user via a multimodal user interface. A set of operations relevant to the target is selected and verbal command suggestions relevant to the selected set of operations and the determined target are generated. At least a portion of the generated verbal command suggestions is provided for presentation in association with the multimodal user interface in one of three interface variants: one that presents command suggestions as a list, one that presents command suggestions using contextual overlay windows, and one that presents command suggestions embedded within the interface. Each of the proposed interface variants facilitates user awareness of verbal commands that are capable of being executed and teaches users how available verbal commands can be invoked. ## Claims ### Claim 1 A computing system comprising: one or more processors; and one or more computer storage media storing computer-useable instructions that, when used by the one or more processors, cause the one or more processors to: select an operation relevant to a determined target associated with a direct-manipulation input of a multimodal interface; generate, using a phrasing template selected based on a quantity of modifiable parameters in the phrasing template, a verbal command suggestion relevant to the selected operation for the determined target; and cause display, via the multimodal interface, of the verbal command suggestion presented adjacent to the determined target associated with the direct-manipulation input of the multimodal interface. ### Claim 2 (depends on 1) The computing system of claim 1, wherein the computer-useable instructions, when used by the one or more processors, further cause the one or more processors to populate at least one modifiable parameter of the phrasing template with a parameter value to generate the verbal command suggestion. ### Claim 3 (depends on 2) The computing system of claim 2, wherein the computer-useable instructions, when used by the one or more processors, cause the one or more processors to populate the at least one modifiable parameter with the parameter value based on at least one of an active state of the target and relevance to a workflow engaged in by a user. ### Claim 4 (depends on 1) The computing system of claim 1, wherein the received direct-manipulation input is one of a touch input, a keyboard input, an eye-tracking input, a gesture input, or a mouse input. ### Claim 5 (depends on 1) The computing system of claim 1, wherein the verbal command suggestion is a natural language command suggestion. ### Claim 6 (depends on 1) The computing system of claim 1, wherein selecting the phrasing template is based on at least one of a type of the direct-manipulation input received and a complexity of the phrasing template. ### Claim 7 (depends on 1) The computing system of claim 1, wherein the determined target comprises an object in an image, a menu, or a button. ### Claim 8 (depends on 1) The computing system of claim 1, wherein the verbal command suggestion corresponds with a prioritized command that aligns with a workflow. ### Claim 9 A computer-implemented method, comprising: identifying a direct-manipulation input of a multimodal interface indicating selection to present a list of available operations and corresponding verbal command suggestions; and causing display, via the multimodal interface, of the list of available operations and a set of verbal command suggestions corresponding with each operation in the list of available operations, wherein, based on contextual information, operations and verbal command suggestions unapplicable to an active state of the multimodal interface are visually deemphasized while displayed. ### Claim 10 (depends on 9) The computer-implemented method of claim 9, wherein an available operation and a corresponding set of verbal command suggestions associated with the available operation are presented as a group within the multimodal interface. ### Claim 11 A computer-implemented method comprising: select an operation relevant to a determined target associated with a direct-manipulation input of a multimodal interface; generate, using a phrasing template selected based on a quantity of modifiable parameters in the phrasing template, a verbal command suggestion relevant to the selected operation for the determined target; and cause display, via the multimodal interface, of the verbal command suggestion presented in association with the determined target corresponding with the direct-manipulation input of the multimodal interface. ### Claim 12 (depends on 11) The computer-implemented method of claim 11, wherein the determined target is within an image, and the verbal command suggestion is presented within a property panel. ### Claim 13 (depends on 12) The computer-implemented method of claim 12, wherein emphasized text corresponds to the verbal command suggestion augmenting a graphical user interface widget in the property panel. ### Claim 14 (depends on 11) The computer-implemented method of claim 11, wherein the determined target is within a property panel, and the verbal command suggestion is presented within the property panel. ### Claim 15 (depends on 11) The computer-implemented method of claim 11, wherein the determined target is a talk button. ### Claim 16 (depends on 15) The computer-implemented method of claim 15, wherein when the determined target is the talk button, a first set of verbal command suggestions corresponding to objects in an image and a second set of verbal command suggestions are embedded within a presented toolbar or property panel. ### Claim 17 (depends on 11) The computer-implemented method of claim 11, further comprising causing display of a feedback region that presents one or more command suggestions in response to an unrecognized verbal input. ### Claim 18 (depends on 17) The computer-implemented method of claim 17, wherein the one or more command suggestions are generated based on inferring a most likely failure type, wherein failure types include a phrasing error, a parameter error, an operation-object mapping error, or an operation recognition error.