# US 12,224,969 B2 · Machine learning-based prediction of message responses using a corpus of response templates > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,224,969 B2 - Document ID: US12224969B2 - Status: Granted - Kind: B2 - Application number: 17/743,553 - Filed: 2022-05-13 - Published: 2023-11-16 - Granted: 2025-02-11 - Earliest claimed: 2022-05-13 - Assignees: Dell Products L.P. - Applicants: Dell Products L.P. - Inventors: João Francisco Homrich Da Jornada; Walter Cristian Bruck Mundstock; Daniel Carvalho Liedke - Web page: https://patentgrep.com/patent/US12224969B2 ## CPC - H04L 51/02: using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages - G06F 18/2178: {based on feedback of a supervisor} - G06F 18/24: Classification techniques - G06Q 30/016: After-sales - G06N 20/00: Machine learning - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Techniques are provided for machine learning-based prediction of message responses using a corpus of response templates. One method comprises obtaining a user message sent by a user; applying the user message to a machine learning model that predicts a response template, from a corpus of response templates, that is responsive to the user message, wherein the machine learning model is trained using training data comprising historical user messages sent by multiple users and agent messages sent by customer service agents, wherein the agent messages: (i) have a similarity score with respect to at least one of the response templates that satisfies one or more similarity criteria, and (ii) are sent in response to a corresponding historical user message; and automatically generating a response to the user message using the predicted response template. ## Claims ### Claim 1 A method, comprising: obtaining at least one user message sent by at least one user; applying the at least one user message of the at least one user to at least one machine learning model that predicts one or more response templates, from a corpus of response templates, that are responsive to the at least one user message, wherein the at least one machine learning model is trained using training data comprising at least one historical user message sent by one or more users and at least one historical agent message sent by at least one customer service agent, wherein the at least one historical agent message: (i) has a similarity score with respect to at least one of the response templates, from the corpus of response templates, that satisfies one or more similarity criteria, (ii) is sent in response to the at least one historical user message sent by the one or more users; and (iii) is based at least in part on one or more of the at least one response template from the corpus of response templates, wherein an identifier of the at least one response template is used as a classification label in a supervised learning of the at least one machine learning model and wherein the at least one response template and one or more respective variations, that satisfy the one or more similarity criteria, of the at least one response template are considered together during the training of the at least one machine learning model; and automatically generating at least one response to the at least one user message using the predicted one or more response templates; wherein the method is performed by at least one processing device comprising a processor coupled to a memory. ### Claim 2 (depends on 1) The method of claim 1, wherein the at least one machine learning model predicts a plurality of the response templates that are responsive to the at least one user message and wherein the plurality of response templates are sorted using a corresponding confidence score associated with the prediction of each of the plurality of the response templates. ### Claim 3 (depends on 2) The method of claim 2, further comprising presenting at least a subset of the plurality of the response templates that are responsive to the at least one user message to the at least one customer service agent, using a graphical user interface, for a selection by the at least one customer service agent to prepare at least one response to the at least one user message. ### Claim 4 (depends on 2) The method of claim 2, wherein a number of the plurality of response templates that are presented to the at least one customer service agent is selected based at least in part on an accuracy of the trained machine learning model determined during a testing phase of the at least one machine learning model. ### Claim 5 (depends on 2) The method of claim 2, wherein the corresponding confidence score is determined within a context of a larger communication comprising the at least one user message. ### Claim 6 (depends on 1) The method of claim 1, wherein the at least one machine learning model is trained to correlate the one or more response templates from the corpus of response templates with a content of the at least one historical user message sent by the one or more users prior to a sending of the corresponding at least one historical agent message. ### Claim 7 (depends on 1) The method of claim 1, further comprising employing user feedback regarding the automatically generated at least one response to update the at least one machine learning model. ### Claim 8 (depends on 1) The method of claim 1, wherein the at least one user message is part of a live chat communication between the at least one user and the at least one customer service agent. ### Claim 9 An apparatus comprising: at least one processing device comprising a processor coupled to a memory; the at least one processing device being configured to implement the following steps: obtaining at least one user message sent by at least one user; applying the at least one user message of the at least one user to at least one machine learning model that predicts one or more response templates, from a corpus of response templates, that are responsive to the at least one user message, wherein the at least one machine learning model is trained using training data comprising at least one historical user message sent by one or more users and at least one historical agent message sent by at least one customer service agent, wherein the at least one historical agent message: (i) has a similarity score with respect to at least one of the response templates, from the corpus of response templates, that satisfies one or more similarity criteria, (ii) is sent in response to the at least one historical user message sent by the one or more users; and (iii) is based at least in part on one or more of the at least one response template from the corpus of response templates, wherein an identifier of the at least one response template is used as a classification label in a supervised learning of the at least one machine learning model and wherein the at least one response template and one or more respective variations, that satisfy the one or more similarity criteria, of the at least one response template are considered together during the training of the at least one machine learning model; and automatically generating at least one response to the at least one user message using the predicted one or more response templates. ### Claim 10 (depends on 9) The apparatus of claim 9, wherein the at least one machine learning model predicts a plurality of the response templates that are responsive to the at least one user message and wherein the plurality of response templates are sorted using a corresponding confidence score associated with the prediction of each of the plurality of the response templates. ### Claim 11 (depends on 10) The apparatus of claim 10, further comprising presenting at least a subset of the plurality of the response templates that are responsive to the at least one user message to the at least one customer service agent, using a graphical user interface, for a selection by the at least one customer service agent to prepare at least one response to the at least one user message. ### Claim 12 (depends on 10) The apparatus of claim 10, wherein the corresponding confidence score is determined within a context of a larger communication comprising the at least one user message. ### Claim 13 (depends on 9) The apparatus of claim 9, wherein the at least one machine learning model is trained to correlate the one or more response templates from the corpus of response templates with a content of the at least one historical user message sent by the one or more users prior to a sending of the corresponding at least one historical agent message. ### Claim 14 (depends on 9) The apparatus of claim 9, further comprising employing user feedback regarding the automatically generated at least one response to update the at least one machine learning model. ### Claim 15 A non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device to perform the following steps: obtaining at least one user message sent by at least one user; applying the at least one user message of the at least one user to at least one machine learning model that predicts one or more response templates, from a corpus of response templates, that are responsive to the at least one user message, wherein the at least one machine learning model is trained using training data comprising at least one historical user message sent by one or more users and at least one historical agent message sent by at least one customer service agent, wherein the at least one historical agent message: (i) has a similarity score with respect to at least one of the response templates, from the corpus of response templates, that satisfies one or more similarity criteria, (ii) is sent in response to the at least one historical user message sent by the one or more users; and (iii) is based at least in part on one or more of the at least one response template from the corpus of response templates, wherein an identifier of the at least one response template is used as a classification label in a supervised learning of the at least one machine learning model and wherein the at least one response template and one or more respective variations, that satisfy the one or more similarity criteria, of the at least one response template are considered together during the training of the at least one machine learning model; and automatically generating at least one response to the at least one user message using the predicted one or more response templates. ### Claim 16 (depends on 15) The non-transitory processor-readable storage medium of claim 15, wherein the at least one machine learning model predicts a plurality of the response templates that are responsive to the at least one user message and wherein the plurality of response templates are sorted using a corresponding confidence score associated with the prediction of each of the plurality of the response templates. ### Claim 17 (depends on 16) The non-transitory processor-readable storage medium of claim 16, further comprising presenting at least a subset of the plurality of the response templates that are responsive to the at least one user message to the at least one customer service agent, using a graphical user interface, for a selection by the at least one customer service agent to prepare at least one response to the at least one user message. ### Claim 18 (depends on 16) The non-transitory processor-readable storage medium of claim 16, wherein the corresponding confidence score is determined within a context of a larger communication comprising the at least one user message. ### Claim 19 (depends on 15) The non-transitory processor-readable storage medium of claim 15, wherein the at least one machine learning model is trained to correlate the one or more response templates from the corpus of response templates with a content of the at least one historical user message sent by the one or more users prior to a sending of the corresponding at least one historical agent message. ### Claim 20 (depends on 15) The non-transitory processor-readable storage medium of claim 15, further comprising employing user feedback regarding the automatically generated at least one response to update the at least one machine learning model.