# US 12,602,710 B2 · Ensemble of language models for improved user support

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,602,710 B2
- Document ID: US12602710B2
- Status: Granted
- Kind: B2
- Application number: 18/510,541
- Filed: 2023-11-15
- Published: 2025-05-15
- Granted: 2026-04-14
- Earliest claimed: 2023-11-15
- Assignees: Intuit Inc.
- Applicants: Intuit, Inc.
- Inventors: Jineet Hiren Doshi; Maya Vered Livshits; Apurva Swarnakar; Jingyuan Zhang
- Web page: https://patentgrep.com/patent/US12602710B2

## CPC

- G06Q 30/0282: Rating or review of business operators or products
- G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30)
- G06F 40/30: Semantic analysis
- G06Q 30/016: After-sales
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Certain aspects of the disclosure provide a method for providing user support by generating recommended response for a customer verbatim with an ensemble of machine learning models. The method includes processing a customer verbatim with a topic model trained to identify a topic associated with the customer verbatim. The method further includes processing the customer verbatim with a sentiment model trained to determine a sentiment of the customer verbatim. The method further includes processing the customer verbatim with an actionability model trained to assign an actionability score to the customer verbatim. The method includes processing the topic, the sentiment, and the actionability score with a recommendation model to generate the recommended response to the customer verbatim.

## Claims

### Claim 1

A method for training an ensemble of machine learning models to provide recommendations for responding to a customer verbatim, comprising:
training through unsupervised learning, with a first training dataset comprising unlabeled data, a topic model to group, using natural language processing, the unlabeled data of the first training dataset based on a common theme, each group associated with a topic, whereby the topic model is trained to parse the customer verbatim using natural language processing and output an indication of a topic of the customer verbatim based on association of the customer verbatim with the common theme;
training through supervised learning, with a second training dataset comprising sets of sentiment labels, each set of the sets of sentiment labels comprising a labeled customer verbatim and a desired determined sentiment, a sentiment model to classify, using natural language processing, the customer verbatim and output an indication of a sentiment of a customer of the customer verbatim, wherein the sentiment indicates a customer's satisfaction based on the customer verbatim, and the sentiment model is trained to reach a minimum of a loss of a loss function by adjusting one or more weights of the sentiment model to reduce the loss;
training through supervised learning, with a third training dataset comprising sets of actionability labels, each set of the sets of actionability labels comprising a labeled customer verbatim and a desired actionability score, an actionability model to classify, using natural language processing, the customer verbatim and output an actionability score for the customer verbatim, wherein the actionability score indicates an estimated resolvability of the customer verbatim by a support service; and
training with reinforcement learning, with a fourth training dataset, a recommendation model to process the output of the topic model, the output of the sentiment model, and the output of the actionability model, and output a recommended response to the customer verbatim, based on the topic, the sentiment, and the actionability score, and the recommendation model is instantiated with a decision matrix capturing prior knowledge.

### Claim 2 (depends on 1)

The method of claim 1, wherein the topic model is trained with one or more of: latent dirichlet allocation, latent semantic analysis, non-negative matrix factorization, or BERTopic modeling.

### Claim 3 (depends on 1)

The method of claim 1, wherein the recommendation model is further trained to indicate a communication channel for a user to contact a customer in regards to the customer verbatim.

### Claim 4 (depends on 1)

The method of claim 1, wherein the topic model is further trained based on domain-specific knowledge associated with a domain of the support service.

### Claim 5 (depends on 1)

The method of claim 1, wherein the topic model is further trained to output a data log indicating a plurality of topics associated with a plurality of customer verbatims by grouping one or more of the plurality of customer verbatims based on a common topic of the plurality of topics.

### Claim 6 (depends on 1)

The method of claim 1, wherein the indication of the sentiment of the customer comprises a positive or negative classification of the customer verbatim.

### Claim 7 (depends on 1)

The method of claim 1, wherein:
the indication of the sentiment of the customer comprises a satisfaction score; and
the sentiment model is further trained based on domain-specific knowledge associated with a domain of the support service.

### Claim 8 (depends on 1)

The method of claim 1, wherein the actionability score further indicates a priority for responding to the customer verbatim.

### Claim 9 (depends on 1)

The method of claim 1, wherein the recommendation model is further trained to indicate a time of day to contact the customer.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
identifying a topic of a subsequent customer verbatim, by:
processing, with the topic model, the subsequent customer verbatim; and
outputting, by the topic model, the topic of the subsequent customer verbatim;
determining a sentiment of the subsequent customer verbatim, by:
processing, with the sentiment model, the subsequent customer verbatim; and
outputting, by the topic model, the sentiment of the subsequent customer verbatim;
assigning an actionability score for the subsequent customer verbatim, by:
processing, with the actionability model, the subsequent customer verbatim; and
outputting, by the topic model, the sentiment of the subsequent customer verbatim; and
generating a recommended response to the subsequent customer verbatim, by:
processing, with the recommendation model, the topic, the sentiment, and the actionability score of the subsequent customer verbatim; and
outputting, the recommended response.

### Claim 11 (depends on 10)

The method of claim 10, wherein:
the recommended response to the subsequent customer verbatim comprises a resolution for the subsequent customer verbatim, and the method further comprises sending, to a user, the resolution for the subsequent customer verbatim.

### Claim 12 (depends on 11)

The method of claim 11, further comprising receiving an indication of activity of the customer associated with the resolution for the subsequent customer verbatim.

### Claim 13 (depends on 12)

The method of claim 12, further comprising updating the recommendation model based on the indication of activity.

### Claim 14

A processing system, comprising: a memory comprising computer-executable instructions; and a processor configured to execute the computer-executable instructions and cause the processing system to:
train through unsupervised learning, with a first training dataset comprising unlabeled data, a topic machine learning model to, group, using natural-language processing techniques, the unlabeled data of the first training dataset based on a common theme, each group associated with a topic, whereby the topic machine learning model is trained to parse a customer verbatim and output an indication of the topic of the customer verbatim based on association of the customer verbatim with the common theme;
train through supervised learning, with a second training dataset comprising sets of sentiment labels, each set of the sets of sentiment labels comprising a labeled customer verbatim and a desired determined sentiment, a sentiment machine learning model to classify, using natural language processing, the customer verbatim and output a indication of a sentiment of a customer of the customer verbatim, wherein the sentiment indicates a customer's satisfaction based on the customer verbatim, and the sentiment machine learning model is trained to reach a minimum of a loss of a loss function by adjusting one or more weights of the sentiment machine learning model to reduce the loss;
train through supervised learning, with a third training dataset comprising sets of actionability labels, each set of the sets of actionability labels comprising a labeled customer verbatim and a desired actionability score, an actionability machine learning model to classify, using natural language processing, the customer verbatim and output an actionability score for the customer verbatim, wherein the actionability score indicates an estimated resolvability of the customer verbatim by a support service; and
train with reinforcement learning, with a fourth training dataset, a recommendation model to process the output of the topic machine learning model, the output of the sentiment machine learning model, and the output of the actionability machine learning model, and output a recommended response based on the topic, the sentiment, and the actionability score, and the recommendation model is instantiated with a decision matrix capturing prior knowledge.

### Claim 15 (depends on 14)

The processing system of claim 14, wherein the topic machine learning model is trained with one or more of: latent dirichlet allocation, latent semantic analysis, non-negative matrix factorization, or BERTopic modeling.

### Claim 16 (depends on 14)

The processing system of claim 14, wherein the recommendation model is further trained to indicate a communication channel for a user to contact a customer in regards to the customer verbatim.

### Claim 17 (depends on 14)

The processing system of claim 14, wherein the topic machine learning model is further trained based on domain-specific knowledge associated with a domain of the support service.

### Claim 18 (depends on 14)

The processing system of claim 14, wherein the topic machine learning model is further trained to output a data log indicating a plurality of topics associated with a plurality of customer verbatims by grouping one or more of the plurality of customer verbatims based on a common topic of the plurality of topics.

### Claim 19 (depends on 14)

The processing system of claim 14, wherein the indication of the sentiment of the customer comprises a positive or negative classification of the customer verbatim.

### Claim 20 (depends on 14)

The processing system of claim 14, wherein:
the indication of the sentiment of the customer comprises a satisfaction score; and
the sentiment machine learning model is further trained based on domain-specific knowledge associated with a domain of the support service.