# US 8,543,704 B2 · Method and apparatus for multimodal voice and web services > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,543,704 B2 - Document ID: US8543704B2 - Status: Granted - Kind: B2 - Application number: 11/910,301 - Filed: 2006-04-06 - Published: 2009-06-04 - Granted: 2013-09-24 - Earliest claimed: 2005-04-08 - Assignees: International Business Machines Corporation - Applicants: Ronald John Bowater; Charles Willard Cross, Jr.; Leslie Robert Wilson; Yuk-Lun Wong - Inventors: Ronald John Bowater; Charles Willard Cross, Jr.; Leslie Robert Wilson; Yuk-Lun Wong - Web page: https://patentgrep.com/patent/US8543704B2 ## CPC - H04M 3/493: Interactive information services, e.g. directory enquiries {; Arrangements therefor, e.g. interactive voice response [IVR] systems or voice portals} - G06F 3/16: Sound input; Sound output (speech processing G10L) - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - H04M 7/006: {Networks other than PSTN/ISDN providing telephone service, e.g. Voice over Internet Protocol (VoIP) , including next generation networks with a packet-switched transport layer (H04L65/00 takes precedence; aspects not specific to the type of network H04M3/00; special services in those networks H04M3/42)} - H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A voice server can be located, temporarily allocated, and sent audio. The results are returned to a voice client, and the voice server is deallocated for use by the next person talking into their client browser. Voice channels and IVR ports are initially set up by a switch and the IVR using conventional audio protocols. The voice channels are not initially connected to the client. The switch handles the allocation and deallocation of IVR voice channels without having to communicate further with the IVR. A user indicates to the client device that he wishes to initiate a voice interaction during an X+V session. This translates to a request on the CTRL channel to synchronise XHTML and VXML forms as a trigger for the VXML browser to execute a conversational turn. A multiplexer intercepts this control command and establishes a virtual voice circuit between the client device and an existing open but unattached voice port. The virtual circuit is established without having to set up an RTP channel. The CTRL signal is then forwarded to an interaction manager so that the conversation can take place. At the end of the conversation the virtual circuit is disconnected. ## Claims ### Claim 1 A method for controlling an audio connection between an audio interface with a client and an audio processor with a remote voice server comprising: setting up a processor link for audio data with the audio processor and a server switch; setting up an audio interface session comprising establishing an interface link for audio data with the audio interface and the server switch; in response to a signal indicating the start of an audio interaction comprising a multimodal dialogue within the audio interface session, connecting, by the server switch, the processor link and the interface link whereby audio data can flow between the audio interface and the audio processor, and wherein the server switch is to buffer and play back the audio data to the audio processor later if all ports on the audio processor are being utilized; in response to a signal indicating the end of the audio interaction comprising synchronization of fields in the multimodal dialogue, terminating the audio interface session by disconnecting, by the server switch, the processor link and the interface link; and in response to the end of the interface session, taking down the interface link. ### Claim 2 (depends on 1) A method according to claim 1 wherein the audio processor is an interactive voice response system for speech recognition. ### Claim 3 (depends on 2) A method according to claim 2 wherein the audio interface is an XML browser with voice functionality for multimodal operation. ### Claim 4 (depends on 3) A method according to claim 3 wherein the buffering and the playing back of the audio data is based on a delay connecting the interface link with the processor link. ### Claim 5 (depends on 4) A method according to claim 4 wherein the setting up of a processor link with the audio processor further comprises negotiating a RTP connection using SIP protocol. ### Claim 6 (depends on 5) The method according to claim 5 wherein the signal indicating the start of an audio interaction is a pre-existing signal. ### Claim 7 (depends on 6) A method according to claim 6 wherein the signal indicating the end of an audio interaction is a pre-existing signal. ### Claim 8 A switch for controlling an audio connection from an audio interface with a client to an audio processor with a remote voice server comprising: means for setting up a processor link for audio data with the audio processor and a server switch; means for setting up an audio interface session comprising establishing an interface link for audio data with the audio interface and the server switch; means for, in response to a signal indicating the start of an audio interaction comprising a multimodal dialogue within the audio interface session, connecting the processor link and the interface link whereby audio data can flow between the audio interface session and the audio processor, and wherein the server switch is to buffer and play back the audio data to the audio processor later if all ports on the audio processor are being utilized; means for, in response to a signal indicating the end of the audio interaction comprising synchronization of fields in a multimodal dialogue, terminating the audio interface session by disconnecting, by the server switch, the processor link and the interface link; and means for, in response to the end of the interface session, taking down the interface link. ### Claim 9 A multimodal web server comprising: an Extensible HyperText Markup Language (XHTML) and voice documents database; an XHTML and voice filter; a Voice Extensible Markup Language (VoiceXML) browser; an interactive voice response server (IVR); a voice server; and a switch for controlling an audio connection from a client audio interface to the IVR, said switch comprising: means for setting up an IVR link for audio data with the IVR; means for setting up an client interface link for audio data with the client audio interface; means for, in response to a signal indicating the start of an audio interaction comprising a multimodal dialogue, connecting the IVR link and the client interface link, whereby audio data can flow between the client audio interface session and the IVR, and wherein the switch is to buffer and play back the audio data later to the IVR if all ports on the IVR are being utilized; means for, in response to a signal indicating the end of the client audio interaction comprising synchronization of fields in the multimodal dialogue, disconnecting, by the switch, the IVR link and the client interface link; and means for, in response to the end of the client interface session, taking down the client interface. ### Claim 10 A computer program product for controlling an audio connection between an audio interface with a client and an audio processor with a remote voice server, said computer program product including a computer readable storage device having computer usable program code embodied therewith, said computer program code comprising: computer usable program code configured to set up a processor link for audio data with the audio processor and a server switch; computer usable program code configured to set up an audio interface session comprising establishing an interface link for audio data with the audio interface and the server switch; computer usable program code configured to, in response to a signal indicating the start of an audio interaction comprising a multimodal dialogue within the audio interface session, connect the processor link and the interface link whereby audio data can flow between the audio interface and the audio processor, and wherein the server switch is to buffer and play back the audio data to the audio processor later if all ports on the audio processor are being utilized; computer usable program code configured to, in response to a signal indicating the end of the audio interaction comprising synchronization of fields in a multimodal dialogue, terminate the audio interface session by disconnecting, by the server switch, the processor link and the interface link; and computer usable program code configured to, in response to the end of the interface session, take down the interface link. ### Claim 11 (depends on 10) A computer program product according to claim 10 wherein the audio processor is an interactive voice response system for speech recognition. ### Claim 12 (depends on 11) A computer program product according to claim 11 wherein the audio interface is an XML browser with voice functionality for multimodal operation. ### Claim 13 (depends on 12) A computer program product according to claim 12 wherein the buffering and the playing back of the audio data is based on a delay connecting the interface link with and processor link. ### Claim 14 (depends on 13) A computer program product according to claim 13 wherein the computer usable program code configured to set up an interface link for audio data with the audio interface further comprises computer usable program code configured to negotiate an RTP connection using SIP protocol. ### Claim 15 (depends on 14) A computer program product according to claim 14 wherein the signal indicating the start of an audio interaction is a pre-existing signal. ### Claim 16 (depends on 15) A computer program product according to claim 15 wherein the signal indicating the end of an audio interaction is a pre-existing signal. ### Claim 17 An apparatus for controlling an audio connection from a client audio interface to a interactive voice response server (IVR), said apparatus comprising: means for setting up an IVR link for audio data with the IVR; means for setting up an client interface link for audio data with the client audio interface; means for, in response to a signal indicating the start of an audio interaction comprising a multimodal dialogue, connecting the IVR link and the client interface link, whereby audio data can flow between the client audio interface session and the IVR, and wherein a server switch is to buffer and play back the audio data to the IVR later if all ports on the IVR are being utilized; means for, in response to a signal indicating the end of the client audio interaction comprising synchronization of fields in the multimodal dialogue, disconnecting, by the server switch, the IVR link and the client interface link; and means for, in response to the end of the client interface session, taking down the client interface.