# US 12,190,906 B1 · Systems and methods for predicting an emotion based on a multimodal input

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,190,906 B1
- Document ID: US12190906B1
- Status: Granted
- Kind: B1
- Application number: 17/645,031
- Filed: 2021-12-17
- Granted: 2025-01-07
- Earliest claimed: 2021-12-17
- Assignees: Wells Fargo Bank, N.A.
- Applicants: Wells Fargo Bank, N.A.
- Inventors: Vinothkumar Venkataraman; Ramesh Babu Sarvesetty; Rahul Ignatius; Naveen Gururaja Yeri; Saurav Kumar
- Web page: https://patentgrep.com/patent/US12190906B1

## CPC

- G10L 25/63: for estimating an emotional state
- G06F 40/30: Semantic analysis
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06N 20/00: Machine learning
- G06V 40/174: {Facial expression recognition}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 15/32: Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
- H04L 51/02: using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
- H04M 3/4936: {Speech interaction details (speech recognition per se G10L15/00)}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)

## Abstract

Systems, apparatuses, methods, and computer program products are disclosed for predicting an emotion in real-time based on a multimodal input including at least (i) an amount of keystrokes over a period of time and (ii) text. An example method may include receiving, by a communications circuitry, a multimodal input from a user including at least (i) an amount of keystrokes over a period of time and (ii) text. The example method may further include generating, by a trained machine learning model of an emotion prediction circuitry and using the multimodal input, an EmotionPrint for the user. The example method may finally include determining, by the emotion prediction circuitry and using the EmotionPrint, a next action.

## Claims

### Claim 1

A method for predicting an emotion based on a multimodal input, the method comprising:
receiving, by a communications circuitry, a multimodal input from a user including at least (i) an amount of keystrokes over a period of time, (ii) text, and (iii) speech;
converting, via automatic speech recognition circuitry, the speech to converted text;
causing, by a speech-context Bidirectional Long Short-Term Memory (BLSTM) of an emotion prediction circuitry and using the text and the converted text, generation of context hidden vectors;
causing, by the emotion prediction circuitry, generation of audio hidden vectors using the speech and an audio BLSTM;
generating, by a trained machine learning model of the emotion prediction circuitry and using the multimodal input, the context hidden vectors, and the audio hidden vectors, an EmotionPrint for the user; and
determining, by the emotion prediction circuitry and using the EmotionPrint, a next action.

### Claim 2 (depends on 1)

The method of claim 1, wherein the multimodal input includes a series of images.

### Claim 3 (depends on 1)

The method of claim 1, wherein generating the EmotionPrint includes:
causing, by a multimodal feature extraction circuitry and using the multimodal input, generation of one or more multimodal features;
extracting, by the multimodal feature extraction circuitry and using the one or more multimodal features, one or more multimodal vectors;
normalizing, via a Softmax layer of an emotion prediction circuitry, the one or more multimodal vectors to form one or more probabilities corresponding to one or more emotions; and
calculating, via the Softmax layer of the emotion prediction circuitry, a probability distribution based on the one or more probabilities corresponding to one or more emotions and wherein the EmotionPrint is generated based on the probability distribution.

### Claim 4 (depends on 3)

The method of claim 3, further comprising, prior to formation of the one or more probabilities corresponding to one or more emotions:
causing, by the emotion prediction circuitry, generation of word aligned hidden vectors using the audio hidden vectors, the context hidden vectors, and an attention layer;
causing, by the emotion prediction circuitry, generation of hidden vectors using the context hidden vectors, the one or more multimodal vectors, and a final BLSTM network;
reducing, via a max-pooling layer, dimensionality of the hidden vectors to generate reduced dimensionality hidden vectors; and
causing, by the emotion prediction circuitry, generation of an emotion vector using the reduced dimensionality hidden vectors and a fully connected layer,
wherein the emotion vector is used to form the one or more probabilities corresponding to one or more emotions.

### Claim 5 (depends on 3)

The method of claim 3, further comprising, prior to generation of one or more multimodal features, pre-processing, by the multimodal feature extraction circuitry, the multimodal input.

### Claim 6 (depends on 5)

The method of claim 5, wherein pre-processing the multimodal input includes applying a corresponding pre-processing algorithm to a corresponding type of input included in the multimodal input.

### Claim 7 (depends on 6)

The method of claim 6, wherein the corresponding pre-processing algorithms include one or more of audio pre-processing algorithms, image pre-processing algorithms, or text pre-processing algorithms.

### Claim 8 (depends on 3)

The method of claim 3, wherein the multimodal input is defined by a portion of communication from a customer.

### Claim 9 (depends on 8)

The method of claim 8, wherein the one or more emotions are determined for each portion of communication from a customer.

### Claim 10 (depends on 3)

The method of claim 3, wherein the one or more multimodal features includes one or more of a number of keystrokes per minute, number of random keystrokes, misspelled words, grammatical errors, or use of punctuations and capitalization.

### Claim 11 (depends on 1)

The method of claim 1, wherein the next action includes providing one or more of personalized product recommendations and personalized service recommendations.

### Claim 12 (depends on 1)

The method of claim 1, wherein the next action includes one or more of transitioning chat based communication to a call, transferring a communication to a live agent, providing a resolution to an issue.

### Claim 13

An apparatus for predicting an emotion based a multimodal input, the apparatus comprising:
a communications circuitry configured to receive a multimodal input from a user including at least (i) an amount of keystrokes over a period of time, (ii) text, and (iii) speech;
an automatic speech recognition circuitry configured to convert the speech to converted text; and
an emotion prediction circuitry configured to:
cause, by a speech-context Bidirectional Long Short-Term Memory (BLSTM) and using the text and the converted text, generation of context hidden vectors,
cause generation of audio hidden vectors using the speech and an audio BLSTM,
generate, by a trained machine learning model using the multimodal input, the context hidden vectors, and the audio hidden vectors, an EmotionPrint for the user, and
determine a next action based on the EmotionPrint.

### Claim 14 (depends on 13)

The apparatus of claim 13, the multimodal input includes text from a communication between an artificial intelligence (AI) chat bot and a customer.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein the next action may include transferring the communication to a call center, and wherein further multimodal inputs include audio based on speech from the customer to an agent or interactive IVR.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the apparatus further comprises:
a call processing circuitry configured to:
execute the next action; and
in response to the next action being a transfer to call center, transfer the EmotionPrint and customer data to a corresponding agent.

### Claim 17 (depends on 13)

The apparatus of claim 13, further comprising:
a multimodal feature extraction circuitry configured to:
cause, using the multimodal input, generation of one or more multimodal features, and
extract, using the one or more multimodal features, one or more multimodal vectors; and
an emotion prediction circuitry configured to:
normalize, via a Softmax layer, the one or more multimodal vectors to form one or more probabilities corresponding to one or more emotions, and
calculate, via the Softmax layer, a probability distribution based on the one or more probabilities corresponding to one or more emotions, wherein the EmotionPrint is generated using the one or more probabilities.

### Claim 18 (depends on 17)

The apparatus of claim 17, wherein the emotion prediction circuitry is further configured to:
cause generation of word aligned hidden vectors using the audio hidden vectors, the context hidden vectors, and an attention layer;
cause generation of hidden vectors using the context hidden vectors, the one or more multimodal vectors, and a final BLSTM network;
reduce, via a max-pooling layer, dimensionality of the hidden vectors to generate reduced dimensionality hidden vectors; and
cause generation of an emotion vector using the reduced dimensionality hidden vectors and a fully connected layer,
wherein the emotion vector is used to form the one or more probabilities corresponding to one or more emotions.

### Claim 19

A computer program product for predicting a customer's emotions, the computer program product comprising at least one non-transitory computer-readable storage medium storing software instructions that, when executed, cause an apparatus to:
receive a multimodal input from a user including at least (i) an amount of keystrokes over a period of time, (ii) text, and (iii) speech;
convert the speech to converted text;
cause, by a speech-context Bidirectional Long Short-Term Memory (BLSTM) and using the text and the converted text, generation of context hidden vectors;
cause generation of audio hidden vectors using the speech and an audio BLSTM;
generate, by a trained machine learning model using the multimodal input, the context hidden vectors, and the audio hidden vectors, an EmotionPrint of the user; and
determine a next action based on the EmotionPrint.

### Claim 20 (depends on 19)

The computer program product of claim 19, wherein generation of the EmotionPrint is based on an average amount of keystrokes per minute, latency of keystrokes, key hold time, average speed of keystrokes, amount of deletes, amount of special characters used, or variability.

### Claim 21 (depends on 20)

The computer program product of claim 20, wherein generation of the EmotionPrint is further based on text features, and wherein the text features include one or more of a term frequency-inverse document frequency, embedding, profanity, or praise.

### Claim 22 (depends on 19)

The computer program product of claim 19, wherein the user is an agent, wherein the agent provides a plurality of multimodal inputs, and wherein an EmotionPrint is determined for each of the plurality of multimodal inputs.

### Claim 23 (depends on 22)

The computer program product of claim 22, wherein each EmotionPrint for each of the plurality of multimodal inputs for the agent are aggregated to define an agent's score.