# US 2022/0036899 A1 · Multi-modal conversational agent platform > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0036899 A1 - Document ID: US20220036899A1 - Status: Published application - Kind: A1 - Application number: 17/500,352 - Filed: 2021-10-13 - Published: 2022-02-03 - Earliest claimed: 2019-11-26 - Assignees: Vui, Inc. - Applicants: Vui, Inc. - Inventors: Giuseppe Di Fabbrizio; Evgeny Stepanov; Giuliano Tortoreto; Mehool Patel; Phil C. Frank - Web page: https://patentgrep.com/patent/US20220036899A1 ## CPC - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G06F 16/3329: Natural language query formulation - G06F 16/36: Creation of semantic tools, e.g. ontology or thesauri - G06F 40/30: Semantic analysis - G06N 20/00: Machine learning - G06Q 30/0603: {Catalogue creation or management} - G06Q 30/0641: {utilising user interfaces specially adapted for shopping} - G10L 13/00: Speech synthesis; Text to speech systems - G10L 15/063: {Training} - G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning} - G06Q 30/016: After-sales - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 2015/223: {Execution procedure of a spoken command} - G10L 2015/228: {of application context} - H04M 3/493: Interactive information services, e.g. directory enquiries {; Arrangements therefor, e.g. interactive voice response [IVR] systems or voice portals} - H04M 2201/405: involving speaker-dependent recognition - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08) ## Abstract A method includes receiving data characterizing an utterance of a query associated with a tenant; providing, to an automated speech recognition engine, the received data and a profile selected from a plurality of profiles based on the tenant, the profile configuring the automated speech recognition engine to process the received data; receiving, from the automated speech recognition engine, a text string characterizing the query; and processing, via an ensemble of natural language agents configured based on the tenant, the text string characterizing the query to determine a textual response to the query, the textual response including at least one word from a first lexicon associated with the tenant. Related systems, methods, apparatus, and computer readable mediums are also described. ## Claims ### Claim 1 A method comprising: receiving, by a multitenant remote server including executable instances of natural language resources, data characterizing a query by a first user and associated with a first tenant, the multitenant remote server including a tenant portal enabling the first user configuration of tenant data; deploying, responsive to the receiving, a first instance of an executable natural language resource configured to receive a text string characterizing the query and determine a textual response to the query; providing, to an automated speech recognition engine via the multitenant remote server, the received data; receiving, from the automated speech recognition engine, a text string characterizing the query; and processing, via the first instance of the executable natural language agent ensemble, the text string characterizing the query to determine a textual response to the query, the textual response including at least one word from a lexicon associated with the first tenant. ### Claim 2 (depends on 1) The method of claim 1, further comprising providing, to a text-to-speech synthesis engine, the textual response; receiving, from the text-to-speech synthesis engine, a verbalized query response determined by the text-to-speech synthesis engine based on the textual response; and providing the verbalized query response. ### Claim 3 (depends on 2) The method of claim 2, wherein the text-to-speech synthesis engine is configured to receive the textual response, and to generate, in response to the receiving, the verbalized query response including audio data corresponding to the received textual response, the text-to-speech synthesis engine being selected from one or more inter-changeable speech processing engines included in the profile. ### Claim 4 (depends on 2) The method of claim 2, further comprising: receiving, prior to receiving data characterizing the query, an input to a web site provided via a web browser configured on a first computing device, the input causing the web browser to be authenticated and registered at a second computing device coupled to the first computing device via a network. ### Claim 5 (depends on 4) The method of claim 4, further comprising: receiving, by the second computing device, validation data associated with the first computing device, the validation data including audio and graphical rendering settings configured on with the first computing device; generating, in response to confirming the validation data, an initial conversation prompt by the second computing device and providing the initial conversation prompt to the web site configured on the first computing device; receiving, at an input device coupled to the first computing device and in response to providing the initial conversation prompt via the web site, the data characterizing an utterance of the query, the query associated with an item available via the web site; transmitting the provided verbalized query response to the first computing device; and providing the verbalized query response via an output device coupled to the first computing device. ### Claim 6 (depends on 1) The method of claim 1, further comprising providing a first configuration of a graphical user interface on a first client device, the client device configured to receive the utterance from a user. ### Claim 7 (depends on 1) The method of claim 1, wherein processing the text string characterizing the query further comprises: generating a sematic interpretation associated with the text string; determining a first contextual sequence associated with text string based on one or more previously processed text strings; generating a first response action based on the determined first contextual sequence; and generating the textual response based on the generated first response action. ### Claim 8 (depends on 7) The method of claim 7, wherein the semantic interpretation is generated using a first data structure representing the lexicon associated with the first tenant. ### Claim 9 (depends on 8) The method of claim 8, wherein the first data structure is generated based on at least one of: a catalog of items associated with the first tenant and including a first item title and a first item description, one or more reviews associated with a first item, interactive user data associated with a first item, or a combination thereof. ### Claim 10 (depends on 9) The method of claim 9, wherein generating the first data structure includes determining one or more attributes associated with a first item from the catalog of items; determining one or more synonyms associated with the first item from the catalog of items; determining one or more referring expressions associated with the first item from the catalog of items and/or the interactive user data associated with the first item; generating the first data structure based on the determining steps, the first data structure including a name, one or more attributes, one or more synonyms, one or more referring expressions, and/or one or more dialogs corresponding to the first item. ### Claim 11 (depends on 8) The method of claim 8, wherein the first data structure is used to train the at least one of a plurality of classification algorithms. ### Claim 12 (depends on 1) The method of claim 1, further comprising: receiving second data characterizing an utterance of a second query associated with a second tenant; providing, to a second automated speech recognition engine, the received second data; receiving, from the automated speech recognition engine, a second text string characterizing the second query; and processing, via a second instance of the natural language agent ensemble configured based on the second tenant, the second text string characterizing the second query to determine a second textual response to the second query, the second textual response including at least one word from a second lexicon associated with the second tenant. ### Claim 13 (depends on 1) The method of claim 1, wherein the query includes a plurality of natural language words spoken by the first user and received by an input device of a first computing device, the query provided by the first user in regard to a first context associated with a first item provided by the first tenant. ### Claim 14 (depends on 1) The method of claim 1, wherein the received data is provided to the automated speech recognition engine with a profile selected from a plurality of profiles based on the first tenant, the profile configuring the automated speech recognition engine to process the received data. ### Claim 15 (depends on 14) The method of claim 14, wherein the profile includes one or more configuration settings associated with the first instance of the natural language agent ensemble configured on a server including a data processor, one or more configuration settings associated with the natural language agent ensemble configured on a first computing device, or one or more configuration settings specifying one or more speech processing engines configured on a server including a data processor. ### Claim 16 (depends on 1) The method of claim 1, wherein the first tenant includes at least one of a retail entity, a service provider entity, a financial entity, a manufacturing entity, an entertainment entity, an information storage entity, and a data processing entity. ### Claim 17 (depends on 1) The method of claim 1, wherein the automated speech recognition engine is configured to receive audio data corresponding to the query and to generate, in response to the receiving, the text string including textual data corresponding to the received audio data, the automatic speech recognition engine being selected from one or more inter-changeable speech processing engines. ### Claim 18 (depends on 1) The method of claim 1, wherein the data characterizing the query associated with the first tenant is provided via a textual interaction modality or via a speech interaction modality. ### Claim 19 A system comprising: at least one data processor; and memory storing instructions, which, when executed by the at least one data processor cause the at least one data processor to perform operations comprising: receiving, by a multitenant remote server including executable instances of natural language resources, data characterizing a query by a first user and associated with a first tenant, the multitenant remote server including a tenant portal enabling the first user configuration of tenant data; deploying, responsive to the receiving, a first instance of an executable natural language resource configured to receive a text string characterizing the query and determine a textual response to the query; providing, to an automated speech recognition engine via the multitenant remote server, the received data; receiving, from the automated speech recognition engine, a text string characterizing the query; and processing, via the first instance of the executable natural language agent ensemble, the text string characterizing the query to determine a textual response to the query, the textual response including at least one word from a lexicon associated with the first tenant. ### Claim 20 (depends on 19) The system of claim 19, the operations further comprising: providing, to a text-to-speech synthesis engine, the textual response; receiving, from the text-to-speech synthesis engine, a verbalized query response determined by the text-to-speech synthesis engine based on the textual response; and providing the verbalized query response.