# US 2021/0082437 A1 · Detecting and recovering out-of-vocabulary words in voice-to-text transcription systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0082437 A1
- Document ID: US20210082437A1
- Status: Published application
- Kind: A1
- Application number: 16/570,876
- Filed: 2019-09-13
- Published: 2021-03-18
- Earliest claimed: 2019-09-13
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: SAMUEL THOMAS; KARTIK AUDHKHASI; ZOLTAN TUESKE; YINGHUI HUANG; MICHAEL ALAN PICHENY
- Web page: https://patentgrep.com/patent/US20210082437A1

## CPC

- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 15/32: Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/16: using artificial neural networks
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/183: using context dependencies, e.g. language models
- G10L 2015/088: {Word spotting}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Aspects of the present disclosure describe techniques for identifying and recovering out-of-vocabulary words in transcripts of a voice data recording using word recognition models and word sub-unit recognition models. An example method generally includes receiving a voice data recording for transcription into a textual representation of the voice data recording. The voice data recording is transcribed into the textual representation using a word recognition model. An unknown word is identified in the textual representation, and the unknown word is reconstructed based on recognition of sub-units of the unknown word generated by a sub-unit recognition model. The textual representation of the voice data recording is modified by replacing the unknown word with the reconstruction of the unknown word, and the modified textual representation is output.

## Claims

### Claim 1

A method for recovering out-of-vocabulary words in transcriptions of a voice data recording, comprising:
receiving a voice data recording for transcription into a textual representation of the voice data recording;
transcribing the voice data recording into the textual representation using a word recognition model;
identifying an unknown word in the textual representation;
reconstructing the unknown word in the textual representation based on recognition of sub-units of the unknown word generated by a sub-unit recognition model;
modifying the textual representation of the voice data recording by replacing the unknown word with the reconstruction of the unknown word; and
outputting the modified textual representation of the voice data recording.

### Claim 2 (depends on 1)

The method of claim 1, wherein:
the word recognition model comprises a connectionist temporal classification model trained using a training data set of whole words and sub-words; and
the sub-unit recognition model comprises a connectionist temporal classification model trained using sub-words.

### Claim 3 (depends on 2)

The method of claim 2, wherein the sub-units comprise phonetic components of words, and wherein reconstructing the unknown word comprises:
synthesizing the recognized sub-units into a single unit bounded by an ending time of a word preceding the unknown word and a beginning time of a word succeeding the unknown word; and
looking up the unknown word using the single unit and a reverse phonetic dictionary.

### Claim 4 (depends on 2)

The method of claim 2, wherein the sub-units comprise pieces from which a word can be constructed, each piece including a beginning of word symbol and end of word symbol, and wherein reconstructing the unknown word comprises combining the recognized sub-units into a single unit by connecting an end of word symbol of each respective sub-unit to a beginning of word symbol of a succeeding sub-unit.

### Claim 5 (depends on 2)

The method of claim 2, wherein identifying the unknown word in the textual representation comprises determining that the word recognition model has output a sub-word unit for a segment in the voice data recording.

### Claim 6 (depends on 2)

The method of claim 2, further comprising adjusting an interpolation weight of the word recognition model to adjust a preference for generating the textual representation using whole words or sub-words.

### Claim 7 (depends on 1)

The method of claim 1, wherein:
the word recognition model comprises a sequence-to-sequence model trained using a training data set of whole words; and
the sub-unit recognition model comprises a sequence-to-sequence model trained using a training data set of sub-words.

### Claim 8 (depends on 7)

The method of claim 7, wherein identifying the unknown word in the textual representation comprises identifying a reserved sub-unit in the textual representation indicating that the word recognition model has identified the unknown word as an out-of-vocabulary word.

### Claim 9 (depends on 7)

The method of claim 7, wherein reconstructing the unknown word comprises:
identifying a starting time and ending time for the unknown word in the voice data recording based on attention peaks of a whitespace sub-unit preceding a time spike associated with the unknown word and a whitespace sub-unit succeeding the time spike associated with the unknown word; and
constructing a word based on sub-units recognized by the sub-unit recognition model at each time spike between the whitespace sub-unit preceding the time spike associated with the unknown word and the whitespace sub-unit succeeding the time spike associated with the unknown word.

### Claim 10

A system, comprising:
a processor; and
a memory having instruction stored thereon which, when executed by the processor, performs an operation for recovering out-of-vocabulary words in transcriptions of a voice data recording, the operation comprising:
receiving a voice data recording for transcription into a textual representation of the voice data recording;
transcribing the voice data recording into the textual representation using a word recognition model;
identifying an unknown word in the textual representation;
reconstructing the unknown word in the textual representation based on recognition of sub-units of the unknown word generated by a sub-unit recognition model;
modifying the textual representation of the voice data recording by replacing the unknown word with the reconstruction of the unknown word; and
outputting the modified textual representation of the voice data recording.

### Claim 11 (depends on 10)

The system of claim 10, wherein:
the word recognition model comprises a connectionist temporal classification model trained using a training data set of whole words and sub-words; and
the sub-unit recognition model comprises a connectionist temporal classification model trained using sub-words.

### Claim 12 (depends on 11)

The system of claim 11, wherein the sub-units comprise phonetic components of words, and wherein reconstructing the unknown word comprises:
synthesizing the recognized sub-units into a single unit bounded by an ending time of a word preceding the unknown word and a beginning time of a word succeeding the unknown word; and
looking up the unknown word using the single unit and a reverse phonetic dictionary.

### Claim 13 (depends on 11)

The system of claim 11, wherein the sub-units comprise pieces from which a word can be constructed, each piece including a beginning of word symbol and end of word symbol, and wherein reconstructing the unknown word comprises combining the recognized sub-units into a single unit by connecting an end of word symbol of each respective sub-unit to a beginning of word symbol of a succeeding sub-unit.

### Claim 14 (depends on 11)

The system of claim 11, wherein identifying the unknown word in the textual representation comprises determining that the word recognition model has output a sub-word unit for a segment in the voice data recording.

### Claim 15 (depends on 11)

The system of claim 11, wherein the operation further comprises adjusting an interpolation weight of the word recognition model to adjust a preference for generating the textual representation using whole words or sub-words.

### Claim 16 (depends on 10)

The system of claim 10, wherein:
the word recognition model comprises a sequence-to-sequence model trained using a training data set of whole words; and
the sub-unit recognition model comprises a sequence-to-sequence model trained using a training data set of sub-words.

### Claim 17 (depends on 16)

The system of claim 16, wherein identifying the unknown word in the textual representation comprises identifying a reserved sub-word in the textual representation indicating that the word recognition model has identified the unknown word as an out-of-vocabulary word.

### Claim 18 (depends on 16)

The system of claim 16, wherein reconstructing the unknown word comprises:
identifying a starting time and ending time for the unknown word in the voice data recording based on attention peaks of a whitespace sub-unit preceding a time spike associated with the unknown word and a whitespace sub-unit succeeding the time spike associated with the unknown word; and
constructing a word based on sub-units recognized by the sub-unit recognition model at each time spike between the whitespace sub-unit preceding the time spike associated with the unknown word and the whitespace sub-unit succeeding the time spike associated with the unknown word.

### Claim 19

A computer-readable medium having instructions stored thereon which, when executed by a processor, performs an operation for recovering out-of-vocabulary words in transcriptions of a voice data recording, the operation comprising:
receiving a voice data recording for transcription into a textual representation of the voice data recording;
transcribing the voice data recording into the textual representation using a word recognition model;
identifying an unknown word in the textual representation;
reconstructing the unknown word in the textual representation based on recognition of sub-units of the unknown word generated by a sub-unit recognition model;
modifying the textual representation of the voice data recording by replacing the unknown word with the reconstruction of the unknown word; and
outputting the modified textual representation of the voice data recording.

### Claim 20 (depends on 19)

The computer-readable medium of claim 19, wherein:
where the word recognition model comprises a connectionist temporal classification model trained using a training data set of whole words and sub-words and the sub-unit recognition model comprises a connectionist temporal classification model trained using sub-words, reconstructing the unknown word comprises:
synthesizing the recognized sub-units of phonetic components of words into a single unit bounded by an ending time of a word preceding the unknown word and a beginning time of a word succeeding the unknown word, and
looking up the unknown word using the single unit and a reverse phonetic dictionary; and
where the word recognition model comprises a sequence-to-sequence model trained using a training data set of whole words and the sub-unit recognition model comprises a sequence-to-sequence model trained using a training data set of sub-words, reconstructing the unknown word comprises:
identifying a starting time and ending time for the unknown word in the voice data recording based on attention peaks of a whitespace sub-unit preceding a time spike associated with the unknown word and a whitespace sub-unit succeeding the time spike associated with the unknown word, and
constructing a word based on sub-units recognized by the sub-unit recognition model at each time spike between the whitespace sub-unit preceding the time spike associated with the unknown word and the whitespace sub-unit succeeding the time spike associated with the unknown word.
