# US 8,788,271 B2 · Controlling user interfaces with contextual voice commands > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,788,271 B2 - Document ID: US8788271B2 - Status: Granted - Kind: B2 - Application number: 11/020,756 - Filed: 2004-12-22 - Published: 2006-06-22 - Granted: 2014-07-22 - Earliest claimed: 2004-12-22 - Assignees: Sap SE - Applicants: Frances James; Rama Gurram - Inventors: Frances James; Rama Gurram - Web page: https://patentgrep.com/patent/US8788271B2 ## CPC - G06F 9/451: Execution arrangements for user interfaces - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract One or more voice-enabled user interfaces include a user interface, and a voice extension module associated with the user interface. The voice extension module is configured to voice-enable the user interface and includes a speech recognition engine, a preprocessor, and an input handler. The preprocessor registers with the speech recognition engine one or more voice commands for signaling for execution of one or more semantic operations that may be performed using a first user interface. The input handler receives a first voice command and communicates with the preprocessor to execute a semantic operation that is indicated by the first voice command. The first voice command is one of the voice commands registered with the speech recognition engine by the preprocessor. ## Claims ### Claim 1 A method for enabling a user interface to be controlled with voice commands, the method comprising: receiving information specifying a first user interface at a client computer system, the first user interface enabling interaction with an existing application executed on an application server and being a standalone interface that is separate from the existing application; constructing one or more semantic operations by processing the information specifying the first user interface on the client computer system, each of the one or more semantic operations including sub-operations that are executable using user interface elements of the first user interface, the sub-operations comprising identifying a user interface element, selecting the user interface element, and inputting data using the user interface element; registering one or more voice commands to enable voice control of the first user interface and for initiating execution of the one or more semantic operations, each voice command corresponding to one of the semantic operations, the application executed on the application server remaining unmodified in view of the constructing one or more semantic operations and the registering one or more voice commands; displaying the first user interface at the client computer system upon registering the one or more voice commands; and performing one of the semantic operations in response to a first voice command, the first voice command not explicitly referencing the user interface element, wherein performing one of the semantic operations includes automatic execution of each of the sub-operations in response to the first voice command. ### Claim 2 (depends on 1) The method of claim 1 further comprising: registering one or more additional voice commands to enable switching to a second user interface of a second application, each additional voice command corresponding to a switch to the second user interface; and performing, in response to a second voice command that is one of the additional voice commands and that corresponds to a switch to the second application, the switch to the second user interface corresponding to the second voice command. ### Claim 3 (depends on 1) The method of claim 1 further comprising: registering one or more additional voice commands to enable voice control of a second user interface, each additional voice command corresponding to a semantic operation that may be performed with the second user interface; and performing one of the semantic operations in response to a second voice command, the second voice command being one of the additional voice commands, the performed semantic operation corresponding to the second voice command. ### Claim 4 (depends on 1) The method of claim 1 further comprising: identifying one or more graphical elements of the first user interface that may be controlled; registering one or more additional voice commands to enable control of the identified graphical elements of the first user interface, each additional voice command corresponding to at least one of the identified graphical elements; and controlling one of the identified graphical elements in response to a second voice command, the second voice command being one of the additional voice commands, the controlled graphical element corresponding to the second voice command. ### Claim 5 (depends on 1) The method of claim 1 further comprising prompting for additional information that further specifies the semantic operation that is performed in response to the first voice command. ### Claim 6 (depends on 1) The method of claim 1 further comprising providing feedback indicating that the semantic operation indicated by the first voice command has been performed. ### Claim 7 (depends on 1) The method of claim 1, wherein the information comprises one or more of a hypertext markup language (HTML) document that identifies user interface elements, a control mechanism implemented by a web browser, and metadata that describes the first user interface and functions provided by the first user interface. ### Claim 8 (depends on 1) The method of claim 1, wherein the first user interface is at least one from a group including a hypertext markup language (HTML) document presented in a web browser, a standalone application, and a user interface for a web services application. ### Claim 9 A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising: receiving information specifying a first user interface at a client computer system, the first user interface enabling interaction with an existing application executed on an application server and being a standalone interface that is separate from the existing application; constructing one or more semantic operations by processing the information specifying the first user interface on the client computer system, each of the one or more semantic operations including sub-operations that are executable using user interface elements of the first user interface, the sub-operations comprising identifying a user interface element, selecting the user interface element, and inputting data using the user interface element; registering one or more voice commands to enable voice control of the first user interface and for initiating execution of the one or more semantic operations, each voice command corresponding to one of the semantic operations, the application executed on the application server remaining unmodified in view of the constructing one or more semantic operations and the registering one or more voice commands; displaying the first user interface at the client computer system upon registering the one or more voice commands; and performing one of the semantic operations in response to a first voice command, the first voice command not explicitly referencing the user interface element, wherein performing one of the semantic operations includes automatic execution of each of the sub-operations in response to the first voice command. ### Claim 10 (depends on 9) The computer storage medium of claim 9, wherein the operations further comprise: registering one or more additional voice commands to enable switching to a second user interface of a second application, each additional voice command corresponding to a switch to the second user interface; and performing, in response to a second voice command that is one of the additional voice commands and that corresponds to a switch to the second application, the switch to the second user interface corresponding to the second voice command. ### Claim 11 (depends on 9) The computer storage medium of claim 9, wherein the operations further comprise: registering one or more additional voice commands to enable voice control of a second user interface, each additional voice command corresponding to a semantic operation that may be performed with the second user interface; and performing one of the semantic operations in response to a second voice command, the second voice command being one of the additional voice commands, the performed semantic operation corresponding to the second voice command. ### Claim 12 (depends on 9) The computer storage medium of claim 9, wherein the operations further comprise: identifying one or more graphical elements of the first user interface that may be controlled; registering one or more additional voice commands to enable control of the identified graphical elements of the first user interface, each additional voice command corresponding to at least one of the identified graphical elements; and controlling one of the identified graphical elements in response to a second voice command, the second voice command being one of the additional voice commands, the controlled graphical element corresponding to the second voice command. ### Claim 13 (depends on 9) The computer storage medium of claim 9, wherein the operations further comprise prompting for additional information that further specifies the semantic operation that is performed in response to the first voice command. ### Claim 14 (depends on 9) The computer storage medium of claim 9, wherein the operations further comprise providing feedback indicating that the semantic operation indicated by the first voice command has been performed. ### Claim 15 (depends on 9) The computer storage medium of claim 9, wherein the information comprises one or more of a hypertext markup language (HTML) document that identifies user interface elements, a control mechanism implemented by a web browser, and metadata that describes the first user interface and functions provided by the first user interface. ### Claim 16 (depends on 9) The computer storage medium of claim 9, wherein the first user interface is at least one from a group including a hypertext markup language (HTML) document presented in a web browser, a standalone application, and a user interface for a web services application. ### Claim 17 A system, comprising: a display; and a computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising: receiving information specifying a first user interface at a client computer system, the first user interface enabling interaction with an existing application executed on an application server and being a standalone interface that is separate from the existing application; constructing one or more semantic operations by processing the information specifying the first user interface on the client computer system, each of the one or more semantic operations including sub-operations that are executable using user interface elements of the first user interface, the sub-operations comprising identifying a user interface element, selecting the user interface element, and inputting data using the user interface element; registering one or more voice commands to enable voice control of the first user interface and for initiating execution of the one or more semantic operations, each voice command corresponding to one of the semantic operations, the application executed on the application server remaining unmodified in view of the constructing one or more semantic operations and the registering one or more voice commands; displaying the first user interface at the client computer system upon registering the one or more voice commands; and performing one of the semantic operations in response to a first voice command, the first voice command not explicitly referencing the user interface element, wherein performing one of the semantic operations includes automatic execution of each of the sub-operations in response to the first voice command. ### Claim 18 (depends on 17) The system of claim 17, wherein the operations further comprise: registering one or more additional voice commands to enable switching to a second user interface of a second application, each additional voice command corresponding to a switch to the second user interface; and performing, in response to a second voice command that is one of the additional voice commands and that corresponds to a switch to the second application, the switch to the second user interface corresponding to the second voice command. ### Claim 19 (depends on 17) The system of claim 17, wherein the operations further comprise: registering one or more additional voice commands to enable voice control of a second user interface, each additional voice command corresponding to a semantic operation that may be performed with the second user interface; and performing one of the semantic operations in response to a second voice command, the second voice command being one of the additional voice commands, the performed semantic operation corresponding to the second voice command. ### Claim 20 (depends on 17) The system of claim 17, wherein the operations further comprise: identifying one or more graphical elements of the first user interface that may be controlled; registering one or more additional voice commands to enable control of the identified graphical elements of the first user interface, each additional voice command corresponding to at least one of the identified graphical elements; and controlling one of the identified graphical elements in response to a second voice command, the second voice command being one of the additional voice commands, the controlled graphical element corresponding to the second voice command.