System and method for multimedia contact center interactions via an audiovisual asynchronous channel

A method of transmitting an interactive audiovisual asynchronous query in a contact center system, the method comprising:

• providing, by a web server, an editable audiovisual interface to a web-enabled user device associated with a first party;
• recording, by the web server, an interaction of the first party with the editable audiovisual interface;
• generating, by the web server, an asynchronous query for the contact center system from the recorded interaction;
• processing, by an audiovisual analysis system, at least one of the audio and the video of the asynchronous query;
• determining, by the audiovisual analysis system, an intent generated from the processed video;
• transmitting, by the audiovisual analysis system, the determined intent and the asynchronous query to a routing server;
• and
• transmitting, by the routing server, a signal for routing the asynchronous query to a second party wherein the determined intent is used to determine the second party recipient of the asynchronous query;
• wherein the determining of intent comprises:
• determining, by the audiovisual analysis system, frames in the asynchronous query indicating an action performed by the first party in video of the query;
• generating, by the audiovisual analysis system, a plurality of audiovisual clips from the query, dividing the query from the determined frames indicating action performed by the first party;
• processing, by the audiovisual analysis system, one of the generated audiovisual clips, wherein the processing comprises using an action recognition application for determining an action performed during the generated audiovisual clip;
• processing, by the audiovisual analysis system, the generated audiovisual clip through an entity recognition application for determining an interacted entity during the generated audiovisual clip;
• processing, by the audiovisual analysis system, the audio of the query that requires stitching to corresponding action identified from audiovisual clips;
• processing, by the audiovisual analysis system, the generated audiovisual clip through a speech recognition application for deriving content of speech from the generated audiovisual clip;
• generating, by the audiovisual analysis system, an audiovisual clip output comprised of determined action, interacted entity, and derived speech content from the processed generated audiovisual clip;
• and
• determining, by the audiovisual analysis system, a first party intent from the generated audiovisual clip output.