US 10,896,428 B1Granted

Dynamic speech to text analysis and contact processing using agent and customer sentiments

AssigneeAmazon Technologies, Inc.

InventorsBalarajan Balasubramaniam, Brian Benjamin Hartley, Mabel Karen Bo Ming Lee, Srdjan Majkic, Simon Gilles Meyffret, Mohammadrasool Raeesi Nafchi, Derek Tong

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

A dynamic contact management system is provided for analyzing aspects of speech from a customer contact and generating dynamic output based on the analysis. The system can process audio data representing utterances that occur during a customer contact, wherein the processing comprises generating scores based on agent and customer sentiments determined using their respective speech to text data. Based on the processing, the system can generate scores for use in dynamically determining which actions to take, updating displays, analyzing contact outcomes over time, etc.

Claims17

Dependent claims

A system comprising:

computer-readable memory storing executable instructions;
and
one or more processors in communication with the computer-readable memory and configured by the executable instructions to at least:
receive a request from a customer device to begin a customer service contact;
determine a contact classification for the customer service contact based at least partly on contact attribute data regarding the customer service contact;
obtain an agent state score representing a sentiment of an agent based on one or more prior customer service contacts assigned to the agent;
prior to beginning the customer service contact, determine to assign the customer service contact to the agent based at least partly on the contact classification and the agent state score;
receive audio data from the customer device, wherein the audio data represents an utterance during the customer service contact;
generate text data representing the utterance using the audio data;
generate sentiment feature data using the text data, wherein the sentiment feature data represents a sentiment associated with the utterance;
generate speech feature data using the audio data, wherein the speech feature data represents an emotion associated with the utterance;
generate a satisfaction score using the sentiment feature data, the speech feature data, and a model trained to predict a degree of satisfaction based at least partly on a sentiment feature and a speech feature;
determine a customer service contact interaction based at least partly on an expected effect on the satisfaction score;
generate graphical user interface data using the satisfaction score and a plurality of previously-generated satisfaction scores, wherein the graphical user interface data represents the customer service contact interaction and a visual summarization of satisfaction scores with respect to time;
send the graphical user interface data to an agent device associated with the customer service contact;
generate, using the satisfaction score, customer service contact evaluation data representing one or more evaluation questions;
send the customer service contact evaluation data to the customer;
receive second audio data from the agent device, wherein the second audio data represents a second utterance of the agent during the customer contact;
generate second text data representing the second utterance using the second audio data;
generate second sentiment feature data using the second text data, wherein the second sentiment feature data represents a sentiment associated with the second utterance;
generate second speech feature data using the second audio data, wherein the second speech feature data represents an emotion associated with the second utterance;
and
generate a second satisfaction score using the second sentiment feature data, the second speech feature data, and the model.

20 more elements

A computer-implemented method comprising:

as performed by a computing system comprising one or more computer processors configured to execute specific instructions,
determining a contact classification for a customer service contact based at least partly on data regarding the customer service contact;
obtaining an agent state score representing a sentiment of an agent based on one or more prior customer service contacts assigned to the agent;
prior to beginning the customer service contact, determining to assign the customer service contact to the agent based at least partly on the contact classification and the agent state score;
receiving audio data representing an utterance associated with the customer service contact;
generating sentiment feature data using the audio data, wherein the sentiment feature data represents a sentiment based at least partly on one or more words of the utterance;
generating speech feature data using the audio data, wherein the speech feature data represents an emotion based at least partly on one or more acoustic features of the utterance;
generating a state score using a model, wherein the state score is generated based on the sentiment feature data and the speech feature data;
determining a customer service contact interaction based at least partly on an expected effect on the customer service contact;
performing an action associated with the customer service contact interaction;
receiving second audio data, wherein the second audio data represents a second utterance of the agent during the customer service contact;
generating second sentiment feature data using the second audio data, wherein the second sentiment feature data represents a sentiment associated with the second utterance;
generating second speech feature data using the second audio data, wherein the second speech feature data represents an emotion associated with the second utterance;
and
generating a second state score associated with the agent using the model, wherein the second state score is generated based on the second sentiment feature data and the second speech feature data.

12 more elements

A system comprising:

computer-readable memory storing executable instructions;
and
one or more processors in communication with the computer-readable memory and configured by the executable instructions to at least:
prior to beginning a customer service contact, determine to assign the customer service contact to an agent based at least partly on an agent state score, wherein the agent state score represents a sentiment of the agent based on one or more prior customer service contacts assigned to the agent;
receive audio data representing an utterance associated with the customer service contact;
generate sentiment feature data using the audio data, wherein the sentiment feature data represents a sentiment associated with the utterance;
generate speech feature data using the audio data, wherein the speech feature data represents an emotion associated with the utterance;
generate a state score using a model, wherein the state score is generated based on the sentiment feature data and the speech feature data;
perform an action associated with the customer service contact based at least partly on the state score;
receive second audio data representing a second utterance associated with the customer service contact, wherein the second utterance is made by the agent;
generate second sentiment feature data using the second audio data, wherein the second sentiment feature data represents a sentiment associated with the second utterance;
generate second speech feature data using the second audio data, wherein the second speech feature data represents an emotion associated with the second utterance;
and
generate a second state score associated with the agent using the model, wherein the second state score is generated based on the second sentiment feature data and the second speech feature data.

11 more elements

Dependencies are inferred from the claim text.