# US 10,372,825 B2 · Emotion detection and expression integration in dialog systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,372,825 B2
- Document ID: US10372825B2
- Status: Granted
- Kind: B2
- Application number: 15/844,877
- Filed: 2017-12-18
- Published: 2019-06-20
- Granted: 2019-08-06
- Earliest claimed: 2017-12-18
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Jonathan Herzig; David Konopnicki; Tommy Sandbank; Michal Shmueli-Scheuer
- Web page: https://patentgrep.com/patent/US10372825B2

## CPC

- G06F 40/30: Semantic analysis
- G06F 40/35: Discourse or dialogue representation
- G10L 25/63: for estimating an emotional state
- G06F 40/295: Named entity recognition
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Utilizing a computing device to detect and respond to emotion in dialog systems. The computing device receives a dialog structure comprising a plurality of dialog nodes. The computing device determines a node emotion level for each of the dialog nodes in the dialog structure based on analysis of one or more intents of each of the dialog nodes in the dialog structure. The computing device determines emotional hotspot nodes in the dialog structure, the node emotion level for each of the emotional hotspot nodes exceeding an emotional threshold. The computing device generates one or more responses modifying the node emotion level of each of the emotional hotspot nodes.

## Claims

### Claim 1

A method of utilizing a computing device for detecting and responding to emotion in dialog systems, the method comprising:
receiving by a computing device a dialog structure, the dialog structure comprising a plurality of dialog nodes;
determining by the computing device a node emotion level for each of the dialog nodes in the dialog structure based on analysis of one or more intents of each of the dialog nodes in the dialog structure;
determining by the computing device emotional hotspot nodes in the dialog structure, the node emotion level for each of the emotional hotspot nodes exceeding an emotional threshold;
generating one or more responses modifying the node emotion level of each of the emotional hotspot nodes;
receiving by the computing device a path traversal through the dialog structure, the path traversal representing a real-time dialog between the automated conversation agent and a dialog user;
determining by an online emotional rating algorithm an adjusted current emotion level for a node in the path traversal, the adjusted current emotion level adjusted for an emotion level of previous nodes in the path traversal;
determining whether the node in the path traversal was determined by the computing device to be one of the emotional hotspot nodes; and
presenting by the computing device one of the one or more responses when responding to the dialog user during the real-time dialog.

### Claim 2 (depends on 1)

The method of claim 1, further comprising transmitting the one or more responses modifying the node emotion level of each of the one or more emotional hotspots to a dialog designer for approval.

### Claim 3 (depends on 2)

The method of claim 2, wherein if the dialog designer rejects all of the one or more responses modifying the node emotion level, the dialog designer adds a dialog designer response created by the dialog designer.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
receiving by the computing device a window size for the online emotional rating algorithm, the window size being a number of sequential nodes in the path traversal accounted for by the online emotional rating algorithm,
wherein the adjusted current emotion level is adjusted for the emotion level of a number of previous nodes in the path traversal according to the window size.

### Claim 5 (depends on 4)

The method of claim 4, wherein the window size is received from the dialog designer.

### Claim 6 (depends on 3)

The method of claim 3, wherein the computing device maintains a variable representing a number of responses presented to a dialog user during the real-time dialog, and previous to presenting the response to the dialog user modifying the emotion level, determining whether the variable representing the number of responses presented to the dialog user during the real-time dialog is not greater than a maximum number of responses during the real-time dialog.

### Claim 7 (depends on 3)

The method of claim 3, wherein previous to the computing device presenting the approved response to the to the dialog user, determining whether a response was transmitted in an immediately previous node in the path traversal and, if not, presenting the approved response.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
updating an analysis of the one or more intents of each of the dialog nodes in the dialog structure and updating a determination of node emotion level for each of the dialog nodes based on the updated analysis of the one or more intents of each of the dialog nodes;
updating a determination of emotional hotspot nodes in the dialog structure, the updated node emotion level for each of the emotional hotspot nodes exceeding an update threshold; and
generating one or more new responses modifying the node emotion level of each of the emotional hotspot nodes.

### Claim 9 (depends on 8)

The method of claim 8, further comprising transmitting the one or more new responses modifying the node emotion level of each of the one or more emotional hotspots to the dialog designer for approval.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
receiving by the computing device a new path traversal through the dialog structure, the new path traversal;
determining by the online emotional rating algorithm a new adjusted current emotion level for the node in the new path traversal, the new adjusted current emotion level adjusted for a new emotion level of previous nodes in the new path traversal;
determining whether the node in the new path traversal was determined by the computing device to be one of the emotional hotspot nodes and whether the dialog designer approved the new response modifying the node emotion level; and
presenting by the computing device one of the one or more new responses modifying the node emotion level when responding to the dialog user during the real-time dialog.

### Claim 11 (depends on 1)

The method of claim 1, wherein the intent of each of the dialog nodes is designed to capture input of the dialog user.

### Claim 12 (depends on 1)

The method of claim 1, wherein the intent of each of the dialog nodes is analyzed by analyzing each sentence of the dialog nodes in an intent-emotions probability vector, averaging the results of each analyzed sentence for the dialog node, and returning the average emotions vector.

### Claim 13 (depends on 1)

The method of claim 1, wherein the computing device presents the one or more responses during the real-time dialog via a text conversation or a speech conversation.

### Claim 14

A computer system for detecting and responding to emotion in dialog systems, the computer system comprising:
one or more computer processors;
one or more computer-readable storage media;
program instructions stored on the computer-readable storage media for execution by at least one of the one or more processors, the program instructions comprising:
program instructions to receive by a computer processor a dialog structure, the dialog structure comprising a plurality of dialog nodes;
program instructions to determine by the computer processor a node emotion level for each of the dialog nodes in the dialog structure based on analysis of one or more intents of each of the dialog nodes in the dialog structure;
program instructions to determine by the computer processor emotional hotspot nodes in the dialog structure, the node emotion level for each of emotional hotspot nodes exceeding an emotional threshold;
program instructions to generate one or more responses modifying the node emotion level of each of the emotional hotspot nodes;
program instructions to receive by the computer processor a path traversal through the dialog structure, the path traversal representing a real-time dialog between the automated conversation agent and a dialog user;
program instructions to determine by the computer processor and adjusted current emotion level for a node in the path traversal, the adjusted current emotion level adjusted for an emotion level of previous nodes in the path traversal;
program instructions to determine whether the node in the path traversal was determined by the computer processor to be one of the emotional hotspots; and
program instructions to present by the computer processor one of the one or more responses when responding to the dialog user during the real-time dialog.

### Claim 15 (depends on 14)

The computer system of claim 14, wherein the program instructions further comprise program instructions to transmit one or more responses modifying the node emotion level of each of the one or more emotional hotspot nodes to a dialog designer for approval.

### Claim 16 (depends on 15)

The computer system of claim 15, wherein if the dialog designer rejects all of the one or more responses modifying the node emotion level, the dialog designer adds a dialog designer response created by the dialog designer.

### Claim 17 (depends on 14)

The computer system of claim 14, wherein the intent of each of the dialog nodes is designed to capture input of the dialog user.

### Claim 18 (depends on 14)

The computer system of claim 14, wherein the intent of each of the dialog nodes in analyzed by analyzing each sentence of the dialog nodes in an intent-emotions probability vector, averaging the results of each analyzed sentence for the dialog node, and returning the average emotions vector.

### Claim 19 (depends on 14)

The computer system of claim 14, wherein the computer processor presents the one or more responses during the real-time dialog via a text conversation or a speech conversation.