# US 2021/0243254 A1 · Reusable multimodal application

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0243254 A1
- Document ID: US20210243254A1
- Status: Published application
- Kind: A1
- Application number: 17/027,512
- Filed: 2020-09-21
- Published: 2021-08-05
- Earliest claimed: 2006-05-05
- Assignees: Gula Consulting Limited Liability Company
- Applicants: Gula Consulting Limited Liability Company
- Inventors: Ewald C. Anderl
- Web page: https://patentgrep.com/patent/US20210243254A1

## CPC

- H04L 67/1095: Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/038: Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 9/451: Execution arrangements for user interfaces
- G06F 16/178: Techniques for file synchronisation in file systems
- G06F 16/27: Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F 16/951: Indexing; Web crawling techniques
- G06F 16/9538: Presentation of query results
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- H04M 3/493: Interactive information services, e.g. directory enquiries {; Arrangements therefor, e.g. interactive voice response [IVR] systems or voice portals}
- H04M 3/4938: {comprising a voice browser which renders and interprets, e.g. VoiceXML}
- G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
- H04M 3/42204: {Arrangements at the exchange for service or number selection by voice (at the terminal H04M1/27)}
- H04M 2201/42: Graphical user interfaces
- H04M 2207/18: wireless networks
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)

## Abstract

A method and system are disclosed herein for accepting multimodal inputs and deriving synchronized and processed information. A reusable multimodal application is provided on the mobile device. A user transmits a multimodal command to the multimodal platform via the mobile network. The one or more modes of communication that are inputted are transmitted to the multimodal platform(s) via the mobile network(s) and thereafter synchronized and processed at the multimodal platform. The synchronized and processed information is transmitted to the multimodal application. if required, the user verifies and appropriately modifies the synchronized and processed information. The verified and modified information are transferred from the multimodal application to the visual application. The final result(s) are derived by inputting the verified and modified results into the visual application.

## Claims

### Claim 1

(canceled)

### Claim 2

A method comprising:
receiving, by a first computing device, a search request from a second computing device via a communications network;
generating, by the first computing device, based on the search request, search information comprising search terms or search results for subsequent verification or selection, wherein a first portion of the search information is for presentation in a first mode of communication and a second portion of the search information is for presentation in a second mode of communication; and
transmitting, by the first computing device, the search information to the second computing device,
wherein the first mode of communication or the second mode of communication comprises a voice mode of communication, and wherein the search information facilitates activation, at the second computing device, of both the first mode of communication and the second mode of communication.

### Claim 3 (depends on 2)

The method of claim 2, wherein the search request is based on input received by the second computing device in a voice mode of communication.

### Claim 4 (depends on 3)

The method of claim 3, wherein the voice mode of communication is activated by a user speaking.

### Claim 5 (depends on 2)

The method of claim 2, wherein the first mode of communication comprises a voice mode, and wherein the second mode of communication comprises a visual mode or a haptic mode.

### Claim 6 (depends on 2)

The method of claim 2, wherein the second mode of communication comprises a voice mode, and wherein the first mode of communication comprises a visual mode or a haptic mode.

### Claim 7 (depends on 2)

The method of claim 2, further comprising:
receiving, by the first computing device, search grammar associated with the search request from the second computing device; and
identifying, by the first computing device, grammar elements associated with the search request and the search grammar.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
determining, by the first computing device, search words based at least in part upon the identified grammar elements; and
including, by the first computing device, the search words in the search information.

### Claim 9 (depends on 2)

The method of claim 2, wherein the search information comprises a plurality of search terms, and wherein the search information facilitates the second computing device rendering the plurality of search terms and permitting user selection among the plurality of search terms.

### Claim 10 (depends on 2)

The method of claim 2, wherein the search information facilitates the second computing device switching from the first mode of communication to the second mode of communication in response to receiving a user selection of the second mode of communication.

### Claim 11 (depends on 2)

The method of claim 2, wherein the second computing device is a mobile computing device.

### Claim 12

A computing device comprising a processor and memory, the memory comprising executable instructions that, in response to being executed by the processor, cause the computing device to:
receive a search request from a remote computing device via a communications network;
generate, based on the search request, search information comprising search terms or search results for subsequent verification or selection, wherein a first portion of the search information is for presentation in a first mode of communication and a second portion of the search information is for presentation in a second mode of communication; and
transmit the search information to the remote computing device,
wherein the first mode of communication or the second mode of communication comprises a voice mode of communication, and wherein the search information facilitates activation, at the remote computing device, of both the first mode of communication and the second mode of communication.

### Claim 13 (depends on 12)

The computing device of claim 12, wherein the search request is based on input received by the remote computing device in a voice mode of communication.

### Claim 14 (depends on 13)

The computing device of claim 13, wherein the voice mode of communication is activated by a user speaking.

### Claim 15 (depends on 12)

The computing device of claim 12, wherein the first mode of communication comprises a voice mode, and wherein the second mode of communication comprises a visual mode or a haptic mode.

### Claim 16 (depends on 12)

The computing device of claim 12, wherein the second mode of communication comprises a voice mode, and wherein the first mode of communication comprises a visual mode or a haptic mode.

### Claim 17 (depends on 12)

The computing device of claim 12, wherein the memory further comprises executable instructions that, in response to being executed by the processor, cause the computing device to:
receive search grammar associated with the search request from the remote computing device; and
identify grammar elements associated with the search request and the search grammar.

### Claim 18 (depends on 17)

The computing device of claim 17, wherein the memory further comprises executable instructions that, in response to being executed by the processor, cause the computing device to:
determine search words based at least in part upon the identified grammar elements; and
include the search words in the search information.

### Claim 19 (depends on 12)

The computing device of claim 12, wherein the search information comprises a plurality of search terms, and wherein the search information facilitates the remote computing device rendering the plurality of search terms and permitting user selection among the plurality of search terms.

### Claim 20 (depends on 12)

The computing device of claim 12, wherein the search information facilitates the second computing device switching from the first mode of communication to the second mode of communication in response to receiving a user selection of the second mode of communication.

### Claim 21 (depends on 12)

The computing device of claim 12, wherein the remote computing device is a mobile computing device.

### Claim 22

A non-transitory, computer-readable storage medium having stored thereon computer-executable instructions comprising instructions that, in response to being executed by a computing device, cause the computing device to:
receive a search request from a remote computing device via a communications network;
generate, based on the search request, search information comprising search terms or search results for subsequent verification or selection, wherein a first portion of the search information is for presentation in a first mode of communication and a second portion of the search information is for presentation in a second mode of communication; and
transmit the search information to the remote computing device,
wherein the first mode of communication or the second mode of communication comprises a voice mode of communication, and wherein the search information facilitates activation, at the remote computing device, of both the first mode of communication and the second mode of communication.

### Claim 23 (depends on 22)

The computer-readable storage medium of claim 22, wherein the search request is based on input received by the remote computing device in a voice mode of communication.

### Claim 24 (depends on 23)

The computer-readable storage medium of claim 23, wherein the voice mode of communication is activated by a user speaking.

### Claim 25 (depends on 22)

The computer-readable storage medium of claim 22, wherein the first mode of communication comprises a voice mode, and wherein the second mode of communication comprises a visual mode or a haptic mode.

### Claim 26 (depends on 22)

The computer-readable storage medium of claim 22, wherein the second mode of communication comprises a voice mode, and wherein the first mode of communication comprises a visual mode or a haptic mode.

### Claim 27 (depends on 22)

The computer-readable storage medium of claim 22, wherein the computer-executable instructions further comprise instructions that, in response to being executed by the computing device, cause the computing device to:
receive search grammar associated with the search request from the remote computing device; and
identify grammar elements associated with the search request and the search grammar.

### Claim 28 (depends on 27)

The computer-readable storage medium of claim 27, wherein the computer-executable instructions further comprise instructions that, in response to being executed by the computing device, cause the computing device to:
determine search words based at least in part upon the identified grammar elements; and
include the search words in the search information.

### Claim 29 (depends on 22)

The computer-readable storage medium of claim 22, wherein the search information comprises a plurality of search terms, and wherein the search information facilitates the remote computing device rendering the plurality of search terms and permitting user selection of one of the plurality of search terms.

### Claim 30 (depends on 22)

The computer-readable storage medium of claim 22, wherein the search information facilitates the remote computing device switching from the first mode of communication to the second mode of communication in response to receiving a user selection of the second mode of communication.

### Claim 31 (depends on 22)

The computer-readable storage medium of claim 22, wherein the remote computing device is a mobile computing device.
