# US 9,520,124 B2 · Discriminative training of document transcription system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,520,124 B2
- Document ID: US9520124B2
- Status: Granted
- Kind: B2
- Application number: 14/942,349
- Filed: 2015-11-16
- Published: 2016-03-17
- Granted: 2016-12-13
- Earliest claimed: 2004-08-20
- Assignees: MModal IP LLC
- Applicants: MModal IP LLC
- Inventors: Lambert Mathias; Girija Yegnanarayanan; Juergen Fritsch
- Web page: https://patentgrep.com/patent/US9520124B2

## CPC

- G10L 15/063: {Training}
- G06F 40/211: Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
- G06F 40/289: Phrasal analysis, e.g. finite state techniques or chunking
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/183: using context dependencies, e.g. language models
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G16H 15/00: ICT specially adapted for medical reports, e.g. generation or transmission thereof
- G10L 15/193: Formal grammars, e.g. finite state automata, context free grammars or word networks
- G10L 2015/0631: {Creating reference templates; Clustering}
- G10L 2015/0633: {using lexical or orthographic knowledge sources}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

A system is provided for training an acoustic model for use in speech recognition. In particular, such a system may be used to perform training based on a spoken audio stream and a non-literal transcript of the spoken audio stream. Such a system may identify text in the non-literal transcript which represents concepts having multiple spoken forms. The system may attempt to identify the actual spoken form in the audio stream which produced the corresponding text in the non-literal transcript, and thereby produce a revised transcript which more accurately represents the spoken audio stream. The revised, and more accurate, transcript may be used to train the acoustic model using discriminative training techniques, thereby producing a better acoustic model than that which would be produced using conventional techniques, which perform training based directly on the original non-literal transcript.

## Claims

### Claim 1

A method for use with a system including a first document containing at least some information in common with a spoken audio stream, the method performed by at least one computer processor executing computer program instructions to perform steps of:
(A) determining that text in the first document represents an instance of a concept, comprising determining that the text has a format associated with the concept;
(B) identifying, based on the identified text and a repository of finite state grammars, a plurality of spoken forms of the concept, including at least one spoken form not contained in the first document, wherein all of the plurality of spoken forms have the same content as each other;
(C) replacing the identified text with a finite state grammar specifying the plurality of spoken forms of the concept to produce a second document;
(D) generating a document-specific language model based on the second document, comprising generating at least some of the document-specific language model based on the finite state grammar;
(E) using the document-specific language model in a speech recognition process to recognize the spoken audio stream and thereby to produce a third document;
(F) filtering text from the third document by reference to the second document to produce a filtered document in which text filtered from the third document is marked as unreliable; and
(G) using the filtered document and the spoken audio stream to train an acoustic model by performing steps of:
(G)(1) applying a first speech recognition process to the spoken audio stream using a set of base acoustic models and a grammar network based on the filtered document to produce a first set of recognition structures;
(G)(2) applying a second speech recognition process to the spoken audio stream using the set of base acoustic models and a second language model to produce a second set of recognition structures; and
(G)(3) performing discriminative training of the acoustic model using the first set of recognition structures, the second set of recognition structures, the filtered document, and only those portions of the spoken audio stream corresponding to text not marked as unreliable in the filtered document.

### Claim 2 (depends on 1)

The method of claim 1, wherein (G) further comprises a step of:
(G)(4) prior to (G)(1), training the set of base acoustic models using the spoken audio stream and the filtered document.

### Claim 3 (depends on 2)

The method of claim 2, wherein step (G)(4) comprises training the set of base acoustic models using maximum likelihood optimization training.

### Claim 4 (depends on 1)

The method of claim 1, wherein the discriminative training comprises maximum mutual information estimation training, wherein the first set of recognition structures comprises a “correct” lattice, and wherein the second set of recognition structures comprises a “general” lattice.

### Claim 5

A non-transitory computer-readable medium comprising computer program instructions executable by at least one computer processor to perform a method for use with a system, the system including a first document containing at least some information in common with a spoken audio stream, the method comprising:
(A) determining that text in the first document represents an instance of a concept, comprising determining that the text has a format associated with the concept;
(B) identifying, based on the identified text and a repository of finite state grammars, a plurality of spoken forms of the concept, including at least one spoken form not contained in the first document, wherein all of the plurality of spoken forms have the same content as each other;
(C) replacing the identified text with a finite state grammar specifying the plurality of spoken forms of the concept to produce a second document, wherein the finite state grammar includes the identified text and text other than the identified text;
(D) generating a document-specific language model based on the second document, comprising generating at least some of the document-specific language model based on the finite state grammar;
(E) using the document-specific language model in a speech recognition process to recognize the spoken audio stream and thereby to produce a third document;
(F) filtering text from the third document by reference to the second document to produce a filtered document in which text filtered from the third document is marked as unreliable; and
(G) using the filtered document and the spoken audio stream to train an acoustic model by performing steps of:
(G)(1) applying a first speech recognition process to the spoken audio stream using a set of base acoustic models and a grammar network based on the filtered document to produce a first set of recognition structures;
(G)(2) applying a second speech recognition process to the spoken audio stream using the set of base acoustic models and a second language model to produce a second set of recognition structures; and
(G)(3) performing discriminative training of the acoustic model using the first set of recognition structures, the second set of recognition structures, the filtered document, and only those portions of the spoken audio stream corresponding to text not marked as unreliable in the filtered document.

### Claim 6 (depends on 5)

The non-transitory computer-readable medium of claim 5, wherein (G) further comprises a step of:
(G)(4) prior to (G)(1), training the set of base acoustic models using the spoken audio stream and the filtered document.

### Claim 7 (depends on 6)

The non-transitory computer-readable medium of claim 6, wherein the step (G)(4) comprises training the set of base acoustic models using maximum likelihood optimization training.

### Claim 8 (depends on 5)

The non-transitory computer-readable medium of claim 5, wherein the discriminative training comprises maximum mutual information estimation training, wherein the first set of recognition structures comprises a “correct” lattice, and wherein the second set of recognition structures comprises a “general” lattice.