# US 11,451,666 B1 · Systems and methods for handling calls based on call insight information

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,451,666 B1
- Document ID: US11451666B1
- Status: Granted
- Kind: B1
- Application number: 17/445,360
- Filed: 2021-08-18
- Granted: 2022-09-20
- Earliest claimed: 2021-08-18
- Assignees: Verizon Patent and Licensing Inc.
- Applicants: Verizon Patent and Licensing Inc.
- Inventors: Subham Biswas; Saurabh Tahiliani
- Web page: https://patentgrep.com/patent/US11451666B1

## CPC

- H04M 3/5183: {Call or contact centers with computer-telephony arrangements}
- G06F 40/35: Discourse or dialogue representation
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 25/06: the extracted parameters being correlation coefficients
- G10L 25/63: for estimating an emotional state
- H04M 3/5175: {Call or contact centers supervision arrangements}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- H04M 2201/40: using speech recognition
- H04M 2203/552: Call annotations
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A device may receive audio data of a first call between a first user and a second user. The device may generate, based on the audio data, time series data associated with an audio signal of the first call and may process, using a first machine learning model, the time series data to generate first call insight information regarding one or more first insights associated with the first call. The device may process the audio data to generate image data associated with the audio signal and may process, using a second machine learning model, the image data to generate second call insight information regarding one or more second insights associated with the first call. The device may combine the first call insight information and the second call insight information to generate combined call insight information and cause an action to be performed based on the combined call insight information.

## Claims

### Claim 1

A method comprising:
receiving, by a device, audio data of a first call between a first user and a second user;
generating, by the device, based on the audio data, time series data associated with an audio signal of the first call;
processing, by the device and using a first machine learning model, the time series data to generate first call insight information regarding one or more first insights associated with the first call;
processing, by the device, the audio data to generate image data associated with the audio signal;
processing, by the device and using a second machine learning model, the image data to generate second call insight information regarding one or more second insights associated with the first call;
combining, by the device, the first call insight information and the second call insight information to generate combined call insight information; and
causing, by the device, an action to be performed based on the combined call insight information,
wherein the action is associated with a second call.

### Claim 2 (depends on 1)

The method of claim 1, wherein causing the action to be performed includes:
causing the second call to be routed to a voicebot; and
controlling an operation of the voicebot during the second call based on a portion of the combined call insight information.

### Claim 3 (depends on 1)

The method of claim 1, wherein causing the action to be performed includes:
causing the second call to be routed to a user device, out of a plurality of user devices that can handle the second call, based on a portion of the combined call insight.

### Claim 4 (depends on 1)

The method of claim 1, wherein the second call is routed to a user device; and
wherein causing the action to be performed includes:
controlling an operation of the user device during the second call.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
processing the combined call insight information, using a dense layer of a neural network, to obtain processed call insight information; and
wherein causing the action to be performed comprises:
causing the action to be performed based on the processed call insight information.

### Claim 6 (depends on 1)

The method of claim 1, wherein causing the action to be performed includes:
monitoring a mental state, of the first user, with respect to handling calls based on a portion of the combined call insight information,
wherein the mental state includes at least one of a measure of stress, a measure of anxiety, or a measure of fatigue; and
preventing the second call from being routed to a user device of the first user based on the mental state of the first user.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
transcribing the first call, based on the audio data, to generate transcription data;
processing, using a third machine learning model, the transcription data to generate third call insight information regarding one or more third insights associated with the first call; and
wherein combining the first call insight information and the second call insight information to generate the combined call insight information includes:
combining the first call insight information, the second call insight information, and the third call insight information to generate the combined call insight information.

### Claim 8

A device, comprising:
one or more processors configured to:
generate, based on audio data of a first call between a first user and a second user, time series data associated with an audio signal of the first call;
process, using a first machine learning model, the time series data to generate first call insight information regarding one or more first insights associated with the first call;
process the audio data to generate image data associated with the audio signal,
wherein the image data identifies at least one of a waveform of the audio signal or a spectrogram of the audio signal;
process, using a second machine learning model, the image data to generate second call insight information regarding one or more second insights associated with the first call; and
cause an action to be performed based on the first call insight information and the second call insight information,
wherein the action is performed during the first call or during a second call subsequent to the first call.

### Claim 9 (depends on 8)

The device of claim 8, wherein the one or more processors are further configured to:
perform a data augmentation operation on the time series data to obtain augmented time series data;
wherein the one or more processors, to perform the data augmentation operation, are configured to at least one of:
add white noise to the time series data,
perform an audio shifting operation on the time series data,
perform a pitch adjustment operation on the time series data, or
perform a speed adjustment operation on the time series data; and
wherein, to process the time series data to generate the first call insight information, the one or more processors are further configured to:
process the augmented time series data to generate the first call insight information.

### Claim 10 (depends on 8)

The device of claim 8, wherein the image data identifies at least one of an image of the waveform or an image of the spectrogram; and
wherein the one or more processors are further configured to:
adjust a resolution of the at least one of the image of waveform or the image of the spectrogram to obtain adjusted image data; and
wherein, to process the image data, the one or more processors are further configured to:
process the adjusted image data to generate the second call insight information.

### Claim 11 (depends on 8)

The device of claim 8, wherein the one or more processors are further configured to:
transcribe the first call, based on the audio data, to generate transcription data;
process, using a third machine learning model, the transcription data to generate third call insight information regarding one or more third insights associated with the first call; and
wherein, to cause the action, the one or more processors are configured to:
cause the action to be performed based on the first call insight information, the second call insight information, and the third call insight information.

### Claim 12 (depends on 11)

The device of claim 11, wherein the one or more processors are further configured to:
combine the first call insight information, the second call insight information, and the third call insight information to generate combined call insight information; and
wherein, to cause the action, the one or more processors are configured to:
cause the action to be performed based on the combined call insight information.

### Claim 13 (depends on 12)

The device of claim 12, wherein the combined call insight information includes at least one of:
information regarding a context associated with the first call,
information regarding a sentiment of the first user or a sentiment of the second user during the first call,
information regarding an intent associated with the first call,
information regarding a resolution associated with the first call,
information regarding a measure of sarcasm of the first user or a measure of sarcasm of the second user during the first call,
information regarding a tone of the first user or a tone of the second user during the first call,
information regarding a speed of speech of the first user or a speed of speech of the second user during the first call,
information regarding a dialect of the first user or a dialect of the second user, or
information regarding a measure of fatigue of the first user or a measure of fatigue of the second user during the first call.

### Claim 14 (depends on 8)

The device of claim 8, wherein, to cause the action, the one or more processors are configured to:
generate a profile associated with the second user, based on the first call insight information and the second call insight information;
identify information to be provided to the second user based on the profile; and
cause the information to be provided to the second user during the second call.

### Claim 15

A non-transitory computer-readable medium storing a set of instructions, the set of instructions comprising:
one or more instructions that, when executed by one or more processors of a device, cause the device to:
obtain transcription data generated based on transcribing a first call between a first user and a second user;
process, using a first machine learning model, the transcription data to generate first call insight information regarding one or more first insights associated with the first call;
obtain time series data associated with an audio signal during the first call;
process, using a second machine learning model, the time series data to generate second call insight information regarding one or more second insights associated with the first call;
obtain image data associated with the first call,
wherein the image data identifies at least one a waveform associated with the audio signal or a spectrogram associated with the audio signal;
process, using a third machine learning model, the image data to obtain third call insight information regarding one or more third insights associated with the first call;
combine the first call insight information, the second call insight information, and the third call insight information to generate combined call insight information; and
at least one of:
cause a voicebot to be trained, based on a first portion of the combined call insight information;
cause a second call to be routed to a user device of the first user, based on a second portion of the combined call insight information; or
control an operation of the user device during the first call or the second call, based on a third portion of the combined call insight information.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein one or more of the first machine learning model, the second machine learning model, and the third machine learning model include neural network models.

### Claim 17 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the one or more instructions further cause the device to:
generate a profile associated with the first user, based on the combined call insight information;
wherein the one or more instructions, to cause the second call to be routed to the user device of the first user, further cause the device to:
identify information regarding the second call; and
cause the second call to be routed to the user device of the first user based on the information regarding the second call and the profile.

### Claim 18 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the combined call insight information includes one or more of:
information regarding a measure of anxiety of the first user or a measure of anxiety of the second user,
information regarding a measure of stress of the first user or a measure of stress of the second user,
information regarding a context associated with the first call,
information regarding a sentiment of the first user or a sentiment of the second user during the first call,
information regarding an intent associated with the first call,
information regarding a resolution associated with the first call,
information regarding a measure of sarcasm of the first user or a measure of sarcasm of the second user,
information regarding a tone of the first user or a tone of the second user during the first call,
information regarding a speed of speech of the first user or a speed of speech of the second user,
information regarding a dialect of the first user or a dialect of the second user, or information regarding a measure of fatigue of the first user or a measure of fatigue of the second user.

### Claim 19 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the one or more instructions further cause the device to:
cause a third call to be routed to the voicebot after causing the voicebot to be trained; and
control an operation of the voicebot during the third call.

### Claim 20 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the one or more second insights are associated with discrete portions of the first call, and
wherein the one or more third insights are associated with an entirety of the first call.