# US 2014/0195231 A1 · Reusable multimodal application

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2014/0195231 A1
- Document ID: US20140195231A1
- Status: Published application
- Kind: A1
- Application number: 14/203,093
- Filed: 2014-03-10
- Published: 2014-07-10
- Earliest claimed: 2006-05-05
- Assignees: Waloomba Tech Ltd., L.L.C.
- Applicants: Waloomba Tech Ltd., L.L.C.
- Inventors: Ewald C. Anderl
- Web page: https://patentgrep.com/patent/US20140195231A1

## CPC

- H04L 67/1095: Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 9/451: Execution arrangements for user interfaces
- G06F 16/178: Techniques for file synchronisation in file systems
- G06F 16/27: Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F 16/951: Indexing; Web crawling techniques
- G06F 16/9538: Presentation of query results
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- H04M 3/493: Interactive information services, e.g. directory enquiries {; Arrangements therefor, e.g. interactive voice response [IVR] systems or voice portals}
- H04M 3/4938: {comprising a voice browser which renders and interprets, e.g. VoiceXML}
- G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
- H04M 3/42204: {Arrangements at the exchange for service or number selection by voice (at the terminal H04M1/27)}
- H04M 2201/42: Graphical user interfaces
- H04M 2207/18: wireless networks
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)

## Abstract

A method and system are disclosed herein for accepting multimodal inputs and deriving synchronized and processed information. A reusable multimodal application is provided on the mobile device. A user transmits a multimodal command to the multimodal platform via the mobile network. The one or more modes of communication that are inputted are transmitted to the multimodal platform(s) via the mobile network(s) and thereafter synchronized and processed at the multimodal platform. The synchronized and processed information is transmitted to the multimodal application. If required, the user verifies and appropriately modifies the synchronized and processed information. The verified and modified information are transferred from the multimodal application to the visual application. The final result(s) are derived by inputting the verified and modified results into the visual application.

## Claims

### Claim 1

A method comprising:
processing, by a first computing device, input information for an application executing on the first computing device, the input information received via a first mode of communication;
transmitting, via a communications network, data indicative of the input information to a second computing device;
receiving, from the second computing device, synchronized information generated by synchronizing, by the second computing device, the input information with a second mode of communication, wherein the first computing device is configured to:
switch between the first and second modes of communication based at least in part upon a selection of the first and second modes of communication; and
continue execution of the application when switching between the first and second modes of communication; and
rendering the synchronized information for verification and modification in the second mode of communication while continuing execution of the application on the first computing device.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first and second of modes of communication comprise one of audio, visual, and haptic modes.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
determining that the input information comprises a search request;
transmitting the search request and associated search grammar via the communications network to the second computing device;
receiving search words determined by synchronizing the input information using the second mode;
identifying grammar elements associated with the search request and the associated search grammar; and
determining search words based at least in part upon the identified grammar elements.

### Claim 4 (depends on 1)

The method of claim 1, wherein the rendering is performed by a user interface application.

### Claim 5 (depends on 1)

The method of claim 1, wherein the processing is performed by a multimodal application.

### Claim 6 (depends on 5)

The method of claim 5, wherein the multimodal application is operable to render a search interface.

### Claim 7 (depends on 5)

The method of claim 5, wherein the multimodal application is operable to render an input interface.

### Claim 8 (depends on 1)

The method of claim 1, wherein the input information comprises text information.

### Claim 9 (depends on 1)

The method of claim 1, wherein the input information comprises a uniform resource locator address.

### Claim 10 (depends on 5)

The method of claim 5, wherein the multimodal application is operable to allow users to switch between visual and voice modes.

### Claim 11 (depends on 5)

The method of claim 5, wherein the multimodal application is operable to allow users to simultaneously use voice and visual modes.

### Claim 12 (depends on 5)

The method of claim 5, wherein the multimodal application further comprises global grammar elements that can be invoked by predefined multimodal invocations.

### Claim 13

A computing device comprising a processor and memory, the computing device configured to:
synchronize voice input information received via a communications network from a mobile device; wherein:
the voice input information is input to a multimodal application executing on the mobile device;
the multimodal application is operative to receive inputs in voice and visual modes of communication; and
the multimodal application is operative to switch between the voice and visual modes of communication based at least in part upon a selection of one of the voice and visual modes; and
process the synchronized voice input information and transmitting the processed information to the mobile device for rendering, verification, and modification of the processed information in the selected one of the voice and visual input modes during continued execution of the multimodal application, wherein the multimodal application is operative to use the processed voice input information for continued execution of the multimodal application using the voice and visual modes of communication.

### Claim 14 (depends on 13)

The computing device of claim 13, wherein the computing device is further configured to:
receive a search request and associated search grammar via the communications network; and
identify grammar elements associated with the search request and the associated search grammar.

### Claim 15 (depends on 14)

The computing device of claim 14, wherein the computing device is further configured to determine search words based at least in part upon the identified grammar elements.

### Claim 16

A non-transitory computer readable storage medium having computer executable instructions stored thereon, the instructions comprising:
instructions for, in response to an indication that a voice input mode is to be used for receiving inputs to an application executing on a mobile device, executing a voice input function on the mobile device;
instructions for transmitting data indicative of voice input information to a multimodal platform via a communications network, the multimodal platform executing on at least one computing device;
instructions for receiving, via the communications network, processed information operable for inputting to the executing application, the processed information generated using the data indicative of the voice input information;
instructions for rendering the received processed information for verification and modification; and
instructions for inputting the verified and modified processed information to the executing application, wherein the mobile device is configured to switch between the voice input mode and another mode of communication based at least in part upon a user selection of the voice input mode and the another mode of communication and to continue execution of the executing application when switching between the modes of communication.

### Claim 17 (depends on 16)

The computer readable storage medium of claim 16, wherein the rendering comprises displaying the received processed information on a display device.

### Claim 18 (depends on 16)

The computer readable storage medium of claim 16, wherein the verification and modification is performed by receiving inputs received via a user input mechanism on the mobile device.

### Claim 19 (depends on 16)

The computer readable storage medium of claim 16, wherein the processed information comprises one or more voice-to-text conversions of the voice input information.

### Claim 20 (depends on 16)

The computer readable storage medium of claim 16, further comprising:
instructions for determining that the voice input information comprises a search request;
instructions for transmitting the search request and associated search grammar via the communications network to the multimodal platform;
instructions for receiving search words determined by synchronizing the input information using the another mode;
instructions for identifying grammar elements associated with the search request and the associated search grammar; and
instructions for determining search words based at least in part upon the identified grammar elements.