# US 2019/0156835 A1 · Diarization Driven by Meta-Information Identified in Discussion Content

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0156835 A1
- Document ID: US20190156835A1
- Status: Published application
- Kind: A1
- Application number: 15/819,158
- Filed: 2017-11-21
- Published: 2019-05-23
- Earliest claimed: 2017-11-21
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Kenneth W. Church; Dimitrios B. Dimitriadis; Petr Fousek; Miroslav Novak; George A. Saon
- Web page: https://patentgrep.com/patent/US20190156835A1

## CPC

- G10L 17/00: Speaker identification or verification techniques
- G06F 40/35: Discourse or dialogue representation
- G10L 15/183: using context dependencies, e.g. language models
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G10L 17/06: Decision making techniques; Pattern matching strategies
- G10L 25/51: for comparison or discrimination
- G10L 17/04: Training, enrolment or model building
- G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An approach is provided that receives an audio stream and utilizes a voice activation detection (VAD) process to create a digital audio stream of voices from at least two different speakers. An automatic speech recognition (ASR) process is applied to the digital stream with the ASR process resulting in the spoken words to which a speaker turn detection (STD) process is applied to identify a number of speaker segments with each speaker segment ending at a word boundary. The STD process analyzes a number of speaker segments using a language model that determines when speaker changes occur. A speaker clustering algorithm is then applied to the speaker segments to associate one of the speakers with each of the speaker segments.

## Claims

### Claim 1

A method implemented by an information handling system that includes a memory and a processor, the method comprising:
receiving an audio stream that comprises both a plurality of speech segments corresponding to a plurality of human speakers and a plurality of non-verbal segments;
utilizing a voice activation detection (VAD) process on the audio stream, wherein an output of the VAD process is a digital audio stream of voices corresponding to the plurality of speech segments;
applying an automatic speech recognition (ASR) process to the digital stream, wherein the ASR process results in a plurality of spoken words;
inputting the VAD process output into an automatic speech recognition (ASR) process, wherein an output of the ASR process comprises a plurality of spoken words corresponding to the plurality of speech segments and is devoid of the plurality of non-verbal segments
inputting the ASR process output to a speaker turn detection (STD) process to the plurality of spoken words, wherein a plurality of speaker segments of contiguous words are selected from the plurality of spoken words and analyzed by a language model that determines when a plurality of speaker changes occur; and
applying a speaker clustering algorithm to the plurality of speaker segments, wherein the speaker clustering algorithm associates an identifier of one of the human speakers with each of the speaker segments.

### Claim 2 (depends on 1)

The method of claim 1 further comprising:
associating a first word from the plurality of spoken words to a first set of vocal qualities;
identifying a second word from the plurality of spoken words that is successive to the first word and corresponds to a second set of vocal qualities;
inserting a speaker change mark between the first word and the second word in response to determining that the first set of vocal qualities is different from the second set of vocal qualities;
analyzing, during the STD process, a selected one of the speaker segments corresponding to the first word based on a set of previous speaker segments, wherein the selected speaker segment is a question;
calculating a speaker change value based on the language model analysis; and
in response to determining that a speaker change occurs based on the speaker change value, maintaining the speaker change mark between the first word and the second word.

### Claim 3 (depends on 2)

The method of claim 2 further comprising:
increasing the speaker change value in response to the language model analysis revealing that the selected speaker segment is a statement;
increasing the speaker change value in response to the language model analysis revealing that the selected speaker segment is a reply; and
decreasing the speaker change value in response to the language model analysis revealing that the selected speaker segment is a continuation of one or more of the previous speaker segments.

### Claim 4 (depends on 1)

The method of claim 1 further comprising:
appending the selected speaker segment to the set of previous speaker segments;
selecting a next one of the speaker segments;
analyzing the selected next speaker segment during the STD process based on the set of previous speaker segment that now includes the selected speaker segment;
analyzing a second speaker change value based on the language model analysis; and
determining whether a second speaker change occurs based on the second speaker change value.

### Claim 5 (depends on 1)

The method of claim 1 further comprising:
identifying a plurality of sets of vocal qualities from the audio stream, wherein each of the sets of vocal qualities corresponds to a different one of the plurality of human speakers;
comparing the plurality of sets of vocal qualities to each of the plurality of spoken words; and
associating one of the human speakers to each of the words based on the comparison.

### Claim 6 (depends on 5)

The method of claim 5 further comprising:
identifying the speaker change occurs;
failing to detect a corresponding speaker change based on the sets of vocal qualities from the audio stream; and
reanalyzing a location of the identified speaker change occurrence with respect to a vocal quality change found between two successive words at the location; and
determining whether the speaker change occurred at the location based on the reanalysis.

### Claim 7 (depends on 1)

The method of claim 1 further comprising:
generating a transcript of the audio stream that includes the plurality of speaker segments and an association of each of the speaker segments to one of the human speakers; and
ingesting the transcript into a corpus utilized by a question answering (QA) system.

### Claim 8

An information handling system comprising:
one or more processors;
a memory coupled to at least one of the processors; and
a set of computer program instructions stored in the memory and executed by at least one of the processors in order to perform actions of:
receiving an audio stream that comprises both a plurality of speech segments corresponding to a plurality of human speakers and a plurality of non-verbal segments;
utilizing a voice activation detection (VAD) process on the audio stream, wherein an output of the VAD process is a digital audio stream of voices corresponding to the plurality of speech segments;
applying an automatic speech recognition (ASR) process to the digital stream, wherein the ASR process results in a plurality of spoken words;
inputting the VAD process output into an automatic speech recognition (ASR) process, wherein an output of the ASR process comprises a plurality of spoken words corresponding to the plurality of speech segments and is devoid of the plurality of non-verbal segments
inputting the ASR process output to a speaker turn detection (STD) process to the plurality of spoken words, wherein a plurality of speaker segments of contiguous words are selected from the plurality of spoken words and analyzed by a language model that determines when a plurality of speaker changes occur; and
applying a speaker clustering algorithm to the plurality of speaker segments, wherein the speaker clustering algorithm associates an identifier of one of the human speakers with each of the speaker segments.

### Claim 9 (depends on 8)

The information handling system of claim 8 wherein the actions further comprise:
associating a first word from the plurality of spoken words to a first set of vocal qualities;
identifying a second word from the plurality of spoken words that is successive to the first word and corresponds to a second set of vocal qualities;
inserting a speaker change mark between the first word and the second word in response to determining that the first set of vocal qualities is different from the second set of vocal qualities;
analyzing, during the STD process, a selected one of the speaker segments corresponding to the first word based on a set of previous speaker segments, wherein the selected speaker segment is a question;
calculating a speaker change value based on the language model analysis; and
in response to determining that a speaker change occurs based on the speaker change value, maintaining the speaker change mark between the first word and the second word.

### Claim 10 (depends on 9)

The information handling system of claim 9 wherein the actions further comprise:
increasing the speaker change value in response to the language model analysis revealing that the selected speaker segment is a statement;
increasing the speaker change value in response to the language model analysis revealing that the selected speaker segment is a reply; and
decreasing the speaker change value in response to the language model analysis revealing that the selected speaker segment is a continuation of one or more of the previous speaker segments.

### Claim 11 (depends on 8)

The information handling system of claim 8 wherein the actions further comprise:
appending the selected speaker segment to the set of previous speaker segments;
selecting a next one of the speaker segments;
analyzing the selected next speaker segment during the STD process based on the set of previous speaker segment that now includes the selected speaker segment;
analyzing a second speaker change value based on the language model analysis; and
determining whether a second speaker change occurs based on the second speaker change value.

### Claim 12 (depends on 8)

The information handling system of claim 8 wherein the actions further comprise:
identifying a plurality of sets of vocal qualities from the audio stream, wherein each of the sets of vocal qualities corresponds to a different one of the plurality of human speakers;
comparing the plurality of sets of vocal qualities to each of the plurality of spoken words; and
associating one of the human speakers to each of the words based on the comparison.

### Claim 13 (depends on 19)

The information handling system of claim 19 wherein the actions further comprise:
identifying the speaker change occurs;
failing to detect a corresponding speaker change based on the sets of vocal qualities from the audio stream; and
reanalyzing a location of the identified speaker change occurrence with respect to a vocal quality change found between two successive words at the location; and
determining whether the speaker change occurred at the location based on the reanalysis.

### Claim 14 (depends on 8)

The information handling system of claim 8 wherein the actions further comprise:
generating a transcript of the audio stream that includes the plurality of speaker segments and an association of each of the speaker segments to one of the human speakers; and
ingesting the transcript into a corpus utilized by a question answering (QA) system.

### Claim 15

A computer program product stored in a computer readable storage medium, comprising computer program code that, when executed by an information handling system, causes the information handling system to perform actions comprising:
receiving an audio stream that comprises both a plurality of speech segments corresponding to a plurality of human speakers and a plurality of non-verbal segments;
utilizing a voice activation detection (VAD) process on the audio stream, wherein an output of the VAD process is a digital audio stream of voices corresponding to the plurality of speech segments;
applying an automatic speech recognition (ASR) process to the digital stream, wherein the ASR process results in a plurality of spoken words;
inputting the VAD process output into an automatic speech recognition (ASR) process, wherein an output of the ASR process comprises a plurality of spoken words corresponding to the plurality of speech segments and is devoid of the plurality of non-verbal segments
inputting the ASR process output to a speaker turn detection (STD) process to the plurality of spoken words, wherein a plurality of speaker segments of contiguous words are selected from the plurality of spoken words and analyzed by a language model that determines when a plurality of speaker changes occur; and
applying a speaker clustering algorithm to the plurality of speaker segments, wherein the speaker clustering algorithm associates an identifier of one of the human speakers with each of the speaker segments.

### Claim 16 (depends on 15)

The computer program product of claim 15 wherein the actions further comprise:
associating a first word from the plurality of spoken words to a first set of vocal qualities;
identifying a second word from the plurality of spoken words that is successive to the first word and corresponds to a second set of vocal qualities;
inserting a speaker change mark between the first word and the second word in response to determining that the first set of vocal qualities is different from the second set of vocal qualities;
analyzing, during the STD process, a selected one of the speaker segments corresponding to the first word based on a set of previous speaker segments, wherein the selected speaker segment is a question;
calculating a speaker change value based on the language model analysis; and
in response to determining that a speaker change occurs based on the speaker change value, maintaining the speaker change mark between the first word and the second word.

### Claim 17 (depends on 16)

The computer program product of claim 16 wherein the actions further comprise:
increasing the speaker change value in response to the language model analysis revealing that the selected speaker segment is a statement;
increasing the speaker change value in response to the language model analysis revealing that the selected speaker segment is a reply; and
decreasing the speaker change value in response to the language model analysis revealing that the selected speaker segment is a continuation of one or more of the previous speaker segments.

### Claim 18 (depends on 15)

The computer program product of claim 15 wherein the actions further comprise:
appending the selected speaker segment to the set of previous speaker segments;
selecting a next one of the speaker segments;
analyzing the selected next speaker segment during the STD process based on the set of previous speaker segment that now includes the selected speaker segment;
analyzing a second speaker change value based on the language model analysis; and
determining whether a second speaker change occurs based on the second speaker change value.

### Claim 19 (depends on 15)

The computer program product of claim 15 wherein the actions further comprise:
identifying a plurality of sets of vocal qualities from the audio stream, wherein each of the sets of vocal qualities corresponds to a different one of the plurality of human speakers;
comparing the plurality of sets of vocal qualities to each of the plurality of spoken words; and
associating one of the human speakers to each of the words based on the comparison.

### Claim 20 (depends on 19)

The computer program product of claim 19 wherein the actions further comprise:
identifying the speaker change occurs;
failing to detect a corresponding speaker change based on the sets of vocal qualities from the audio stream; and
reanalyzing a location of the identified speaker change occurrence with respect to a vocal quality change found between two successive words at the location; and
determining whether the speaker change occurred at the location based on the reanalysis.
