System and method for interacting with live agents in an automated call center

A non-transitory computer-readable medium encoded with computer program code that includes processing steps for generating a user interface for use in a dialog system, wherein the processing steps cause a processor-based computer to execute steps comprising:

• processing caller utterances for multiple simultaneous calls between callers and an automated response generator in a dialog system, the processing including translating caller utterances into a series of words and forming a hypothesis as to which words were spoken by the caller;
• providing a user interface for an agent monitoring a dialog between the callers and the automated response generator;
• displaying, in a first display window of the user interface, state information associated with a dialog between a first caller and an automated response generator in the dialog system;
• displaying, in a second display window of the user interface, one or more active slots associated with the current state of the dialog with the first caller with respective values for each slot of the one or more active slots;
• and
• displaying, in a third display window of the user interface, waveform information for recognized first caller utterances in the dialog, the waveform information showing actual waveforms of caller utterances;
• displaying dialog information for other calls of the multiple simultaneous calls in tabbed subwindows, wherein each subwindow contains a respective first display window, second display window and third display window;
• automatically notifying the agent upon recognition of a potential problem indicated by one of:
• a repeat by a caller of a phrase, a pause of a duration exceeding a defined time limit, and an increase in volume by the caller over a defined threshold volume;
• and
• allowing intervention by a live agent in a telephone call between the caller and the automated response generator of the dialog system in the event of the potential problem when a confidence level in the hypothesis formed does not exceed a defined threshold confidence level.

A method comprising:

• processing caller utterances for multiple simultaneous calls between callers and an automated response generator in a dialog system, the processing including translating caller utterances into a series of words and forming a hypothesis as to which words were spoken by the caller;
• providing a user interface for an agent monitoring a dialog between the callers and the automated response generator;
• displaying, in a first display window of the user interface, state information associated with a dialog between a first caller and an automated response generator in the dialog system;
• displaying, in a second display window of the user interface, one or more active slots associated with the current state of the dialog with the first caller with respective values for each slot of the one or more active slots;
• and
• displaying, in a third display window of the user interface, waveform information for recognized caller utterances in the dialog, the waveform information showing actual waveforms of caller utterances;
• displaying dialog information for other calls of the multiple simultaneous calls in tabbed subwindows, wherein each subwindow contains a respective first display window, second display window and third display window;
• automatically notifying the agent upon recognition of a potential problem indicated by one of:
• a repeat by a caller of a phrase, a pause of a duration exceeding a defined time limit, and an increase in volume by the caller over a defined threshold volume;
• and
• allowing intervention by the agent in a telephone call between the caller and the automated response generator of the dialog system in the event of the potential problem when a confidence level in the hypothesis formed does not exceed a defined threshold confidence level.