# US 11,451,664 B2 · Objective training and evaluation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,451,664 B2 - Document ID: US11451664B2 - Status: Granted - Kind: B2 - Application number: 16/662,783 - Filed: 2019-10-24 - Published: 2021-04-29 - Granted: 2022-09-20 - Earliest claimed: 2019-10-24 - Assignees: CVS Pharmacy, Inc. - Applicants: CVS Pharmacy, Inc. - Inventors: Roger A. Caron; Patrick J. Daniher; Christopher K. Hays; Joseph Livingston; Cadesha M. Prawl - Web page: https://patentgrep.com/patent/US11451664B2 ## CPC - H04M 3/5175: {Call or contact centers supervision arrangements} - G06V 40/174: {Facial expression recognition} - G06V 40/176: {Dynamic expression} - G10L 15/083: {Recognition networks (G10L15/142, G10L15/16 take precedence)} - G10L 25/63: for estimating an emotional state - G10L 2015/088: {Word spotting} - H04M 2203/401: Performance feedback - H04M 2203/403: Agent or workforce training - H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A system and method configured to generate a simulated caller dialog including a caller intended issue for a scenario for testing a customer service representative (CSR). A simulated caller dialog is presented to the CSR and a CSR response to the simulated caller dialog is received and includes a CSR interpretation of the caller intended issue to the simulated caller dialog. An understanding determination result based on an intent determination recognition score is generated by an intent determination recognition model is generated in response to a comparison of the CSR interpretation of the caller intended issue matching the caller intended issue in the simulated caller dialog. A CSR score is generated for the scenario based on the understanding determination result. The CSR score is recorded to a database. ## Claims ### Claim 1 A computer-implemented method, comprising: generating a simulated caller dialog for a scenario for testing a customer service representative (CSR), the simulated caller dialog including a caller intended issue specific to the scenario; presenting at least a portion of the simulated caller dialog to the CSR, the portion including the caller intended issue; receiving a CSR response to the at least the portion of the simulated caller dialog, the CSR response including a CSR interpretation of the caller intended issue in the at least the portion of the simulated caller dialog; generating an understanding determination result based on an intent determination recognition score generated by an intent determination recognition model, the understanding determination result indicating whether the CSR in the CSR response correctly or incorrectly identified the caller intended issue, wherein the caller intended issue is known in advance of the receiving the CSR response; in response to the intent determination recognition score indicating the CSR incorrectly identified the caller intended issue, generating a CSR response suggestion and displaying the CSR response suggestion to the CSR based on the CSR response, the CSR response suggestion including a behavioral indication to increase empathy in the CSR response to a subsequent restatement of the at least the portion of the simulated caller dialog; generating a CSR score for the scenario based on the understanding determination result and an empathy recognition score generated by an empathy recognition model, the empathy recognition model coupled to receive the CSR response, the empathy recognition score generated at least in part based on an application of keyword recognition to words in the CSR response to the simulated caller dialog; and recording the CSR score in a database. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising: generating accolade feedback to the CSR in response to the understanding determination result indicating the CSR correctly identified the caller intended issue. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising: restating the at least the portion of the simulated caller dialog in response to the understanding determination result indicating the CSR incorrectly identified the caller intended issue in the CSR response. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, further comprising: ceasing restating the at least the portion of the simulated caller dialog in response to the understanding determination result indicating the CSR incorrectly identified the caller intended issue for a failure count threshold quantity of occurrences. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the generating the understanding determination result using the intent determination recognition model includes: comparing words in the CSR response to the caller intended issue with words in the caller intended issue in the at least the portion of the simulated caller dialog; and generating the intent determination recognition score in response to an outcome of the comparison. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, further comprising: generating a facial emotional recognition score using a facial emotional recognition model, the facial emotional recognition score in response to the facial emotional recognition model determining a level of emotional recognition in the CSR response to the at least the portion of the simulated caller dialog; and augmenting the CSR score based on the facial emotional recognition score. ### Claim 7 (depends on 5) The computer-implemented method of claim 5, further comprising: generating an empathy keyword usage recognition score using an empathy keyword recognition model, the empathy keyword usage recognition score in response to the empathy keyword recognition model determining a level of empathy keyword usage in the CSR response to the at least the portion of the simulated caller dialog; and augmenting the CSR score based on the empathy keyword usage recognition score. ### Claim 8 (depends on 5) The computer-implemented method of claim 5, further comprising: generating a customer service expected keyword usage recognition score using an expected customer service keyword recognition model, the customer service expected keyword usage recognition score in response to the expected customer service keyword recognition model determining a level of expected customer service keyword usage in the CSR response to the at least the portion of the simulated caller dialog; and augmenting the CSR score based on the customer service expected keyword usage recognition score. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the generating the CSR score further comprises: generating a leaderboard displaying the CSR score of the CSR with respect to previous scores of other CSRs. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the caller intended issue relates to one of a member, patient, or service provider. ### Claim 11 A system, comprising: one or more processors: and a memory storing instructions that, when executed, cause the one or more processors to: generate a simulated caller dialog for a scenario for testing a customer service representative (CSR), the simulated caller dialog including a caller intended issue specific to the scenario; present at least a portion of the simulated caller dialog to the CSR, the portion including the caller intended issue; receive a CSR response to the at least the portion of the simulated caller dialog, the CSR response including a CSR interpretation of the caller intended issue in the at least the portion of the simulated caller dialog; generate an understanding determination result based on an intent determination recognition score generated by an intent determination recognition model, the understanding determination result indicating whether the CSR in the CSR response correctly or incorrectly identified the caller intended issue, wherein the caller intended issue is known in advance of the receiving the CSR response; in response to the intent determination recognition score indicating the CSR incorrectly identified the caller intended issue, generate a CSR response suggestion and displaying the CSR response suggestion to the CSR based on the CSR response, the CSR response suggestion including a behavioral indication to increase empathy in the CSR response to a subsequent restatement of the at least the portion of the simulated caller dialog; generate a CSR score for the scenario based on the understanding determination result and an empathy recognition score generated by an empathy recognition model, the empathy recognition model coupled to receive the CSR response, the empathy recognition score generated at least in part based on an application of keyword recognition to words in the CSR response to the simulated caller dialog; and record the CSR score in a database. ### Claim 12 (depends on 11) The system of claim 11, wherein the instructions further comprise instructions to: generate accolade feedback to the CSR in response to the understanding determination result indicating the CSR correctly identified the caller intended issue. ### Claim 13 (depends on 11) The system of claim 11, wherein the instructions further comprise instructions to: restate the at least the portion of the simulated caller dialog in response to the understanding determination result indicating the CSR incorrectly identified the caller intended issue in the CSR response. ### Claim 14 (depends on 13) The system of claim 13, wherein the instructions further comprise instructions to: cease restating the at least the portion of the simulated caller dialog in response to the understanding determination result indicating the CSR incorrectly identified the caller intended issue for a failure count threshold quantity of occurrences. ### Claim 15 (depends on 11) The system of claim 11, wherein the instructions to compare further includes instructions to cause to: compare words in the CSR response to the caller intended issue with words in the caller intended issue in the at least the portion of the simulated caller dialog; and generate the intent determination recognition score in response to an outcome of the comparison. ### Claim 16 (depends on 15) The system of claim 15, wherein the instructions to generate the understanding determination result using the intent determination recognition model includes further includes instructions to cause to: generate a facial emotional recognition score using a facial emotional recognition model, the facial emotional recognition score in response to the facial emotional recognition model determining a level of emotional recognition in the CSR response to the at least the portion of the simulated caller dialog; and augment the CSR score based on the facial emotional recognition score. ### Claim 17 (depends on 15) The system of claim 15, wherein the instructions further comprise instructions to: generate an empathy keyword usage recognition score using an empathy keyword recognition model, the empathy keyword usage recognition score in response to the empathy keyword recognition model determining a level of empathy keyword usage in the CSR response to the at least the portion of the simulated caller dialog; and augment the CSR score based on the empathy keyword usage recognition score. ### Claim 18 (depends on 15) The system of claim 15, wherein the instructions further comprise instructions to: generate a customer service expected keyword usage recognition score using an expected customer service keyword recognition model, the customer service expected keyword usage recognition score in response to the expected customer service keyword recognition model determining a level of expected customer service keyword usage in the CSR response to the at least the portion of the simulated caller dialog; and augment the CSR score based on the customer service expected keyword usage recognition score. ### Claim 19 (depends on 11) The system of claim 11, wherein the instructions to generate the CSR score further includes instructions to: generate a leaderboard displaying the CSR score of the CSR with respect to previous scores of other CSRs. ### Claim 20 A system, comprising: means for generating a simulated caller dialog for a scenario for testing a customer service representative (CSR), the simulated caller dialog including a caller intended issue specific to the scenario; means for presenting at least a portion of the simulated caller dialog to the CSR, the portion including the caller intended issue; means for receiving a CSR response to the at least the portion of the simulated caller dialog, the CSR response including a CSR interpretation of the caller intended issue in the at least the portion of the simulated caller dialog; means for generating an understanding determination result based on an intent determination recognition score generated by an intent determination recognition model, the understanding determination result indicating whether the CSR in the CSR response correctly or incorrectly identified the caller intended issue, wherein the caller intended issue is known in advance of the receiving the CSR response; in response to the intent determination recognition score indicating the CSR incorrectly identified the caller intended issue, means for generating a CSR response suggestion and displaying the CSR response suggestion to the CSR based on the CSR response, the CSR response suggestion including a behavioral indication to increase empathy in the CSR response to a subsequent restatement of the at least the portion of the simulated caller dialog; means for generating a CSR score for the scenario based on the understanding determination result and an empathy recognition score generated by an empathy recognition model, the empathy recognition model coupled to receive the CSR response, the empathy recognition score generated at least in part based on an application of keyword recognition to words in the CSR response to the simulated caller dialog; and means for recording the CSR score in a database.