# US 2019/0356779 A1 · System and Method for Monitoring and Visualizing Emotions in Call Center Dialogs at Call Centers > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0356779 A1 - Document ID: US20190356779A1 - Status: Published application - Kind: A1 - Application number: 16/531,514 - Filed: 2019-08-05 - Published: 2019-11-21 - Earliest claimed: 2016-11-02 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Thomas G. Zimmerman - Web page: https://patentgrep.com/patent/US20190356779A1 ## CPC - H04M 3/5175: {Call or contact centers supervision arrangements} - G06T 11/26: Drawing of charts or graphs - G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning} - G10L 21/10: Transforming into visible information - G10L 25/63: for estimating an emotional state - H04M 3/5133: {Operator terminal details} - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 2015/088: {Word spotting} - H04M 3/2281: {Call monitoring, e.g. for law enforcement purposes; Call tracing; Detection or prevention of malicious calls} - H04M 2201/40: using speech recognition - H04M 2201/41: using speaker recognition - H04M 2203/2038: Call context notifications - H04M 2203/357: Autocues for dialog assistance - H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A method, system, and a computer program product are provided for monitoring an emotional state of a conversation by monitoring a communication between first and second persons to determine an emotional state of the communication, providing a visual representation of the emotional state of the communication for display to a person having access to the communication, and performing an intervention action to improve the emotional state of the communication in response to detecting the emotional state of the communication meets one or more specified intervention criteria. ## Claims ### Claim at position 1 1-20. (canceled) ### Claim 21 A method of determining an emotional state of a communication to support an intervention, the method comprising: monitoring, by an information handling system comprising a processor and a memory, a communication between a first person and a second person to determine an emotional state of the communication, wherein the emotional state of the communications is selected from a group consisting of anger, frustration, anxiety, tired, suffering and sadness; providing, by the information handling system, a visual representation of the emotional state of the communication for display to a third person, where the visual representation comprises a call state indicator bar comprising a detected dialog word from the communication that is superimposed on a colored bar having a length and color corresponding to the emotional state of the communication; and performing, by the third party, an intervention action to improve the emotional state of the communication between the first person and second person in response to detecting from the visual representation that the emotional state of the communication meets one or more specified intervention criteria by training the second person to avoid a similar communication with a different customer. ### Claim 22 (depends on 21) The method of claim 21, where providing the visual representation comprises projecting, by the information handling system, valence and arousal emotion measurements of the emotional state into a two-dimensional emotional vector and then computing a one-dimensional angle measure of the emotional vector to represent the emotional state, wherein the length and/or color of the colored bar is computed from the one-dimensional vector angle. ### Claim 23 (depends on 21) The method of claim 21, where the first person is a human customer, where the second person is a human or a machine call center agent, and where the third person is a call center supervisor. ### Claim 24 (depends on 21) The method of claim 21, further comprising monitoring, by a second information handling system comprising a processor and a memory, the communication between the first person and second person to determine the emotional state of the communication by analyzing textual content in the communication for emotion content to determine valence and arousal emotion measurements for the textual content. ### Claim 25 (depends on 24) The method of claim 24, where monitoring the communication comprises monitoring a call center dialog between a customer and call center agent by: identifying, by the second information handling system, textual content in the call center dialog; and analyzing, by the second information handling system, the textual content for emotion content using a semantic model to evaluate valence and arousal emotion measurements for the textual content. ### Claim 26 (depends on 21) The method of claim 21, where performing the intervention action comprises performing an intervention action selected from the group consisting of: training the second person to avoid a similar communication with a different customer; notifying the second person or a supervisor for the second person; replacing the second person with a third person; and providing the second person with one or more conversational suggestions to help guide the communication towards a desired outcome. ### Claim 27 (depends on 21) The method of claim 21, where providing the visual representation comprises displaying, by the information handling system, a timeline of instantaneous emotional states for the communication between of the first person and second person. ### Claim 28 (depends on 21) The method of claim 21, where the timeline includes instantaneous emotional interaction values associated with the instantaneous emotional states of the communication between the first person and second person. ### Claim 29 (depends on 27) The method of claim 27, where the timeline includes associated latent emotional interaction values for the first person which are derived from the instantaneous emotional states of the communication between the first person and second person. ### Claim 30 A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on an information handling system, causes the system to determine an emotional state of a communication to support an intervention by: monitoring a communication between a first person and a second person to determine an emotional state of the communication, wherein the emotional state of the communications is selected from a group consisting of anger, frustration, anxiety, tired, suffering and sadness; providing a visual representation of the emotional state of the communication for display to a third person, where the visual representation comprises a call state indicator bar comprising a detected dialog word from the communication that is superimposed on a colored bar having a length and color corresponding to the emotional state of the communication; and performing an intervention action to improve the emotional state of the communication between the first person and second person in response to detecting from the visual representation that the emotional state of the communication meets one or more specified intervention criteria by training the second person to avoid a similar communication with a different customer. ### Claim 31 (depends on 30) The computer program product of claim 30, where providing the visual representation comprises projecting valence and arousal emotion measurements of the emotional state into a two-dimensional emotional vector and then computing a one-dimensional angle measure of the emotional vector to represent the emotional state, wherein the length and/or color of the colored bar is computed from the one-dimensional vector angle. ### Claim 32 (depends on 30) The computer program product of claim 30, where the first person is a human customer, where the second person is a human or a machine call center agent, and where the third person is a call center supervisor. ### Claim 33 (depends on 30) The computer program product of claim 30, wherein the computer readable program, when executed on the system, causes the system to perform the intervention action by performing an intervention action selected from the group consisting of: training the second person to avoid a similar communication with a different customer; notifying the second person or a supervisor for the second person; replacing the second person with a third person; and providing the second person with one or more conversational suggestions to help guide the communication towards a desired outcome. ### Claim 34 (depends on 30) The computer program product of claim 30, where providing the visual representation comprises displaying a timeline of instantaneous emotional states for the communication between of the first person and second person. ### Claim 35 (depends on 34) The computer program product of claim 34, where the timeline includes instantaneous emotional interaction values associated with the instantaneous emotional states of the communication between the first person and second person. ### Claim 36 (depends on 34) The computer program product of claim 34, where the timeline includes associated latent emotional interaction values for the first person which are derived from the instantaneous emotional states of the communication between the first person and second person. ### Claim 37 An information handling system comprising: one or more processors; a memory coupled to at least one of the processors; a set of instructions stored in the memory and executed by at least one of the processors to determine an emotional state of a communication to support an intervention, wherein the set of instructions are executable to perform actions of: monitoring, by the system, a communication between a first person and a second person to determine an emotional state of the communication, wherein the emotional state of the communications is selected from a group consisting of anger, frustration, anxiety, tired, suffering and sadness; providing, by the system, a visual representation of the emotional state of the communication for display to a third person, where the visual representation comprises a call state indicator bar comprising a detected dialog word from the communication that is superimposed on a colored bar having a length and color corresponding to the emotional state of the communication; and performing an intervention action to improve the emotional state of the communication between the first person and second person in response to detecting from the visual representation that the emotional state of the communication meets one or more specified intervention criteria by training the second person to avoid a similar communication with a different customer. ### Claim 38 (depends on 37) The information handling system of claim 37, where providing the visual representation comprises projecting, by the system, valence and arousal emotion measurements of the emotional state into a two-dimensional emotional vector and then computing a one-dimensional angle measure of the emotional vector to represent the emotional state, wherein the length and/or color of the colored bar is computed from the one-dimensional vector angle. ### Claim 39 (depends on 37) The information handling system of claim 37, where providing the visual representation comprises displaying a timeline of instantaneous emotional states for the communication between of the first person and second person. ### Claim 40 (depends on 39) The information handling system of claim 39, where the timeline includes instantaneous emotional interaction values associated with the instantaneous emotional states of the communication between the first person and second person. ### Claim 41 (depends on 39) The information handling system of claim 39, where the timeline includes associated latent emotional interaction values for the first person which are derived from the instantaneous emotional states of the communication between the first person and second person.