# US 11,308,145 B2 · System and method for multimedia contact center interactions via an audiovisual asynchronous channel > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,145 B2 - Document ID: US11308145B2 - Status: Granted - Kind: B2 - Application number: 16/420,258 - Filed: 2019-05-23 - Published: 2020-11-26 - Granted: 2022-04-19 - Earliest claimed: 2019-05-23 - Applicants: Genesys Telecommunications Laboratories, Inc. - Inventors: Archana Sekar; Ravikumar Gopal; Padmapriya Murali; Vidhyasimhan Jayaraghavendran - Web page: https://patentgrep.com/patent/US11308145B2 ## CPC - G06F 16/438: Presentation of query results - G06Q 10/06311: {Scheduling, planning or task assignment for a person or group} - G06F 16/433: {using audio data} - G06F 16/434: {using image data, e.g. images, photos, pictures taken by a user} - G06V 10/82: using neural networks - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - H04M 3/5232: {Call distribution algorithms} - G06V 20/44: {Event detection} - H04M 2201/42: Graphical user interfaces - H04M 2203/2033: Call handling or answering restrictions, e.g. specified by the calling party - H04M 2203/2038: Call context notifications - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08) ## Abstract A system and method for multimedia contact center interactions via an audiovisual asynchronous messaging channel is provided. In an embodiment, an organization might operate a website comprising information and for providing an entry point of communication with a contact center. A communications channel may be integrated directly with the organization's website, which includes an editable audiovisual interface. A user may interact with the editable interface through a variety of means, including sketching on a webpage and recording to create a video form of audiovisual asynchronous query over the web page. The interaction is analyzed and routed to an agent for handling. The agent may then be connected to the user through the asynchronous channel. ## Claims ### Claim 1 A method of transmitting an interactive audiovisual asynchronous query in a contact center system, the method comprising: providing, by a web server, an editable audiovisual interface to a web-enabled user device associated with a first party; recording, by the web server, an interaction of the first party with the editable audiovisual interface; generating, by the web server, an asynchronous query for the contact center system from the recorded interaction; processing, by an audiovisual analysis system, at least one of the audio and the video of the asynchronous query; determining, by the audiovisual analysis system, an intent generated from the processed video; transmitting, by the audiovisual analysis system, the determined intent and the asynchronous query to a routing server; and transmitting, by the routing server, a signal for routing the asynchronous query to a second party wherein the determined intent is used to determine the second party recipient of the asynchronous query; wherein the second party responds to the recorded interaction though the editable audiovisual interface, the responding by the second party comprising: providing, by a web server, an editable audiovisual interface to the second party; recording, by the web server, the second party interaction with the editable audiovisual interface; recording, by the audiovisual analysis system, the second party interaction for preservation by the contact center; and transmitting, by the audiovisual analysis system, the second party interaction to the first party through the asynchronous channel. ### Claim 2 (depends on 1) The method of claim 1, wherein the editable audiovisual interface comprises tools to enable the first party to perform over a web page of the user device one or more of: draw, point, write text, and input speech. ### Claim 3 (depends on 1) The method of claim 1, wherein the processing, further comprises determining an action performed during the asynchronous query by the first party. ### Claim 4 (depends on 1) The method of claim 1, wherein the processing further comprises determining an object engaged on a web page of the user device during the asynchronous query. ### Claim 5 (depends on 1) The method of claim 1, wherein the processing further comprises determining portions of audio needing stitched to corresponding video of the asynchronous query. ### Claim 6 (depends on 1) The method of claim 1, wherein the second party comprises a virtual agent. ### Claim 7 (depends on 1) The method of claim 1, wherein the intent comprises routing parameters, wherein the routing parameters comprise at least one of: customer type, webpage type, intent type, user intent, keyword, mood, priority, and emotion. ### Claim 8 A method of transmitting an interactive audiovisual asynchronous query in a contact center system, the method comprising: providing, by a web server, an editable audiovisual interface to a web-enabled user device associated with a first party; recording, by the web server, an interaction of the first party with the editable audiovisual interface; generating, by the web server, an asynchronous query for the contact center system from the recorded interaction; processing, by an audiovisual analysis system, at least one of the audio and the video of the asynchronous query; determining, by the audiovisual analysis system, an intent generated from the processed video; transmitting, by the audiovisual analysis system, the determined intent and the asynchronous query to a routing server; and transmitting, by the routing server, a signal for routing the asynchronous query to a second party wherein the determined intent is used to determine the second party recipient of the asynchronous query; wherein the determining of intent comprises: determining, by the audiovisual analysis system, frames in the asynchronous query indicating an action performed by the first party in video of the query; generating, by the audiovisual analysis system, a plurality of audiovisual clips from the query, dividing the query from the determined frames indicating action performed by the first party; processing, by the audiovisual analysis system, one of the generated audiovisual clips, wherein the processing comprises using an action recognition application for determining an action performed during the generated audiovisual clip; processing, by the audiovisual analysis system, the generated audiovisual clip through an entity recognition application for determining an interacted entity during the generated audiovisual clip; processing, by the audiovisual analysis system, the audio of the query that requires stitching to corresponding action identified from audiovisual clips; processing, by the audiovisual analysis system, the generated audiovisual clip through a speech recognition application for deriving content of speech from the generated audiovisual clip; generating, by the audiovisual analysis system, an audiovisual clip output comprised of determined action, interacted entity, and derived speech content from the processed generated audiovisual clip; and determining, by the audiovisual analysis system, a first party intent from the generated audiovisual clip output. ### Claim 9 (depends on 8) The method of claim 8, wherein the action performed during the generated audiovisual clip is a drawing by the first party in the video of the query. ### Claim 10 (depends on 8) The method of claim 8, wherein the action performed during the generated audiovisual clip is a typed text by the first party in the video of the query. ### Claim 11 (depends on 8) The method of claim 8, wherein the interacted entity is a web browser object in a web browser window. ### Claim 12 (depends on 8) The method of claim 8, wherein the first party intent is expressed as a JSON object. ### Claim 13 (depends on 8) The method of claim 8, further comprising responding to the first party, wherein the responding comprises an asynchronous automated response. ### Claim 14 (depends on 13) The method of claim 13, wherein the responding to the first party via the asynchronous automated response further comprises the steps of: providing, by a web server, an editable audiovisual interface to the first party; providing, by the web server, an interaction point for the first party; providing, by the web server, a recommendation window in response to the first party interacting with the interaction point; receiving, by the web server, a signal indicating a selection by the first party from the recommendation window; determining, by a recommendation engine, whether an asynchronous automated response exists for the selection by the first party from the recommendation window; and transmitting, by the web server, to the first party the asynchronous automated response if the asynchronous automated response exists for the selection by the first part from the recommendation window, otherwise, routing the interaction to a second party for handling. ### Claim 15 (depends on 14) The method of claim 14, wherein the interaction point is associated with a web browser object shown on the web-enabled end user device. ### Claim 16 (depends on 14) The method of claim 14, wherein the recommendation engine generates the selection from the recommendation window from a database of intents and entity classification generated from other queries.