# US 11,310,365 B2 · Agent action ranking system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,310,365 B2
- Document ID: US11310365B2
- Status: Granted
- Kind: B2
- Application number: 17/306,285
- Filed: 2021-05-03
- Published: 2021-08-19
- Granted: 2022-04-19
- Earliest claimed: 2019-06-18
- Assignees: Express Scripts Strategie Development, Inc.
- Applicants: Express Scripts Strategic Development, Inc.
- Inventors: Christopher M. Myers; Danielle L. Smith
- Web page: https://patentgrep.com/patent/US11310365B2

## CPC

- H04M 3/5183: {Call or contact centers with computer-telephony arrangements}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06Q 30/016: After-sales
- G10L 15/16: using artificial neural networks
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 25/63: for estimating an emotional state
- G10L 25/90: Pitch determination of speech signals
- H04M 3/527: Centralised call answering arrangements not requiring operator intervention
- G06F 16/22: Indexing; Data structures therefor; Storage structures
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 25/30: using neural networks
- G10L 25/48: specially adapted for particular use
- G10L 2015/223: {Execution procedure of a spoken command}
- H04M 2203/255: comprising a personalized user interface
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Method starts with processing, by processor, receiving audio signal of communication session between member-related client device and agent client device. Processor processes audio signal to generate caller utterances, each including audio caller utterance and transcribed caller utterance. Caller utterances includes first and second caller utterances. For each of the plurality of caller utterances, processor generates relationship data based on transcribed caller utterance, generates identified task based on transcribed caller utterance, and generates task completion probability result based on audio caller utterance, relationship data, and identified task, and stores the task completion probability result in a database. Processor computes agent action ranking score that is based on difference between task completion probability result of first caller utterance and task completion probability result of second caller utterance that precedes first caller utterance. Processor generates agent action result including the agent action ranking score. Other embodiments are disclosed herein.

## Claims

### Claim 1

An audio processing system comprising:
a processor; and
a memory component having instructions stored thereon, when executed by the processor, causes the processor to perform operations comprising:
processing an audio signal to generate a plurality of utterances, the plurality of utterances includes an incoming utterance and a transcribed utterance, the plurality of utterances includes at least a first utterance and a second utterance;
for the plurality of utterances,
generating a relationship data using the transcribed utterance,
generating an identified task using the transcribed utterance, and
generating a task completion probability result using the incoming utterance, the relationship data, and the identified task, and
storing the task completion probability result in a database; and
computing an agent action ranking score using a difference between the task completion probability result of the first utterance and the task completion probability result of the second utterance, wherein the second utterance is at a different time than the first utterance.

### Claim 2 (depends on 1)

The audio processing system of claim 1, wherein computing an agent action ranking score includes normalizing the agent action ranking score.

### Claim 3 (depends on 1)

The audio processing system of claim 1, wherein the agent action ranking score indicates a level of beneficial change or a level of negative change based on the difference between the task completion probability result of the first utterance and the task completion probability result of the second utterance.

### Claim 4 (depends on 1)

The audio processing system of claim 1, wherein the first utterance and the second utterance are audio caller utterances to a call center agent device.

### Claim 5 (depends on 4)

The audio processing system of claim 4, wherein the relationship data includes at least one potential relationship between a caller and a patient that is the subject of the transcribed utterance.

### Claim 6 (depends on 1)

The audio processing system of claim 1, wherein the processor is configured to perform processing the audio signal through a flow neural network and a task neural network.

### Claim 7 (depends on 6)

The audio processing system of claim 6, wherein the task neural network is a convolution neural network, and the plurality of flow neural networks are convolution neural networks.

### Claim 8 (depends on 6)

The audio processing system of claim 6, wherein the processor is configured to perform processing the transcribed utterance using the task neural network generate the identified task.

### Claim 9 (depends on 8)

The audio processing system of claim 8, wherein the processor is configured to perform processing the transcribed utterance using the flow neural network to generate a position in a logical flow associated with the identified task.

### Claim 10 (depends on 9)

The audio processing system of claim 9, wherein the processor is configured to perform generating a task result that includes the identified task and the position in the logical flow associated with the identified task.

### Claim 11 (depends on 1)

The system of claim 1, wherein the processor generating the task completion probability result based on the audio utterance, the relationship data, and the identified task is further caused to:
sampling the audio utterance to generate a plurality of samples of the audio utterance;
generating a loudness result based on loudness values of the plurality of samples using a loudness neural network associated with the identified task;
generating a pitch result based on pitch values of the plurality of samples using a pitch neural network associated with the identified task;
generating a tone result for a plurality of words in the transcribed utterance using a tone neural network associated with the identified task; and
generating the task completion probability result using a task completion probability neural network associated with the identified task, wherein generating the task completion probability result is based on the loudness result, the pitch result, or the tone result.

### Claim 12

A computerized method for processing audio utterances, comprising:
processing, using an audio processor, an audio signal to generate a plurality of utterances, the plurality of utterances includes an incoming utterance and a transcribed utterance, the plurality of utterances includes at least a first utterance and a second utterance;
generating, for the plurality of utterances, a relationship data using the transcribed utterance;
generating, for the plurality of utterances, an identified task using the transcribed utterance;
generating a task completion probability result using the incoming utterance, the relationship data, and the identified task;
storing the task completion probability result in a machine readable database; and
computing, using a processor, an agent action ranking score using a difference between the task completion probability result of the first utterance and the task completion probability result of the second utterance, wherein the second utterance is at a different time than the first utterance.

### Claim 13 (depends on 12)

The method of claim 12, wherein computing an agent action ranking score includes normalizing the agent action ranking score.

### Claim 14 (depends on 13)

The method of claim 13, wherein the agent action ranking score indicates a level of beneficial change or a level of negative change based on the difference between the task completion probability result of the first utterance and the task completion probability result of the second utterance.

### Claim 15 (depends on 12)

The method of claim 12, wherein the first utterance and the second utterance are audio caller utterances to a call center agent device.

### Claim 16 (depends on 15)

The method of claim 15, wherein the relationship data includes at least one potential relationship between a caller and a patient that is the subject of the transcribed utterance.

### Claim 17 (depends on 12)

The method of claim 12, wherein processing and generating steps are performed a flow neural network and a task neural network.

### Claim 18 (depends on 12)

The method of claim 12, wherein generating a task result includes generating the identified task and a position in a logical flow associated with the identified task.

### Claim 19 (depends on 12)

The method of claim 12, wherein generating the task completion probability result comprises:
sampling the audio utterance to generate a plurality of samples of the audio utterance;
generating a loudness result based on loudness values of the plurality of samples using a loudness neural network associated with the identified task;
generating a pitch result based on pitch values of the plurality of samples using a pitch neural network associated with the identified task;
generating a tone result for a plurality of words in the transcribed utterance using a tone neural network associated with the identified task; and
generating the task completion probability result using a task completion probability neural network associated with the identified task, wherein generating the task completion probability result is based on the loudness result, the pitch result, or the tone result.

### Claim 20 (depends on 12)

The method of claim 12, wherein processing, using an audio processor, an audio signal to generate a plurality of utterances includes processing an audio record to generate the plurality of utterances and storing the plurality of utterances from a call in a memory operably connected to the audio processor;
wherein the first utterance and the second utterance are audio caller utterances to a call center agent device;
wherein the relationship data includes at least one potential relationship between a caller and a patient that is the subject of the transcribed utterance; and
wherein generating a task completion probability result includes generating the task completion probability result using a neural network.