# US 11,908,454 B2 · Integrating text inputs for training and adapting neural network transducer ASR models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,908,454 B2 - Document ID: US11908454B2 - Status: Granted - Kind: B2 - Application number: 17/539,752 - Filed: 2021-12-01 - Published: 2023-06-01 - Granted: 2024-02-20 - Earliest claimed: 2021-12-01 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Samuel Thomas; Hong-Kwang Kuo; Brian E. D. Kingsbury; George Andrei Saon; Gakuto Kurata - Web page: https://patentgrep.com/patent/US11908454B2 ## CPC - G10L 15/063: {Training} - G06N 3/08: Learning methods - G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit - G10L 15/16: using artificial neural networks - G10L 21/10: Transforming into visible information - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A processor-implemented method trains an automatic speech recognition system using speech data and text data. A computer device receives speech data, and generates a spectrogram based on the speech data. The computing device receives text data associated with an entire corpus of text data, and generates a textogram based upon the text data. The computing device trains an automatic speech recognition system using the spectrogram and the textogram. ## Claims ### Claim 1 A method of using a computing device to train an automatic speech recognition system using speech data and text data, the method comprising: receiving, by a computing device, speech data; generating, by the computing device, a spectrogram based on the speech data; receiving, by the computing device, text data associated with an entire corpus of text data; generating, by the computing device, a textogram based upon the text data; and training, by the computing device, an automatic speech recognition system using the spectrogram and the textogram. ### Claim 2 (depends on 1) The method of claim 1, further comprising: adapting, by the computing device, an automatic speech recognition model into a spoken language understanding model using only text data without a corresponding spectrogram; and utilizing, by the computing device, the spoken language understanding model to interpret the speech data. ### Claim 3 (depends on 1) The method of claim 1, wherein the speech data is first speech data, and wherein the method further comprises: modifying, by the computing device, the automatic speech recognition model with a second speech data that is different from the first speech data. ### Claim 4 (depends on 1) The method of claim 1, wherein the speech data is first speech data, and wherein the method further comprises: modifying, by the computing device, the automatic speech recognition model with text data from a second speech data that is different from the first speech data. ### Claim 5 (depends on 1) The method of claim 1, wherein the speech data is in a first speech language, and wherein the method further comprises: modifying, by the computing device, the automatic speech recognition model with text data from a second speech language that is different from the first speech language. ### Claim 6 (depends on 1) The method of claim 1, wherein the automatic speech recognition system is based on an automatic speech recognition model of the speech data, and wherein the method further comprises: modifying, by the computing device, the automatic speech recognition model using independent text data that is different from text data associated with the speech data. ### Claim 7 A computer program product for training an automatic speech recognition system using speech data and text data, wherein the computer program product comprises a non-transitory computer readable storage device having program instructions embodied therewith, the program instructions readable and executable by a computer to perform a method comprising: receiving speech data; generating a spectrogram based on the speech data; receiving text data associated with an entire corpus of text data; generating a textogram based upon the text data; and training an automatic speech recognition system using the spectrogram and the textogram. ### Claim 8 (depends on 7) The computer program product of claim 7, wherein the method further comprises: adapting an automatic speech recognition model into a spoken language understanding model using only text data without a corresponding spectrogram; and utilizing the spoken language understanding model to interpret the speech data. ### Claim 9 (depends on 7) The computer program product of claim 7, wherein the speech data is first speech data, and wherein the method further comprises: modifying the automatic speech recognition model with a second speech data that is different from the first speech data. ### Claim 10 (depends on 7) The computer program product of claim 7, wherein the speech data is first speech data, and wherein the method further comprises: modifying the automatic speech recognition model with text data from a second speech data that is different from the first speech data. ### Claim 11 (depends on 7) The computer program product of claim 7, wherein the speech data is in a first speech language, and wherein the method further comprises: modifying the automatic speech recognition model with text data from a second speech language that is different from the first speech language. ### Claim 12 (depends on 7) The computer program product of claim 7, wherein the automatic speech recognition system is based on an automatic speech recognition model of the speech data, and wherein the method further comprises: modifying the automatic speech recognition model using independent text data that is different from text data associated with the speech data. ### Claim 13 (depends on 7) The computer program product of claim 7, wherein the program code is provided as a service in a cloud environment. ### Claim 14 A computer system comprising one or more processors, one or more computer readable memories, and one or more computer readable non-transitory storage mediums, and program instructions stored on at least one of the one or more computer readable non-transitory storage mediums for execution by at least one of the one or more processors via at least one of the one or more computer readable memories, the stored program instructions executed to perform a method comprising: receiving speech data; generating a spectrogram based on the speech data; receiving text data associated with an entire corpus of text data; generating a textogram based upon the text data; and training an automatic speech recognition system using the spectrogram and the textogram. ### Claim 15 (depends on 14) The computer system of claim 14, wherein the method further comprises: adapting an automatic speech recognition model into a spoken language understanding model using only text data without a corresponding spectrogram; and utilizing the spoken language understanding model to interpret the speech data. ### Claim 16 (depends on 14) The computer system of claim 14, wherein the speech data is first speech data, and wherein the method further comprises: modifying the automatic speech recognition model with a second speech data that is different from the first speech data. ### Claim 17 (depends on 14) The computer system of claim 14, wherein the speech data is first speech data, and wherein the method further comprises: modifying the automatic speech recognition model with text data from a second speech data that is different from the first speech data. ### Claim 18 (depends on 14) The computer system of claim 14, wherein the speech data is in a first speech language, and wherein the method further comprises: modifying the automatic speech recognition model with text data from a second speech language that is different from the first speech language. ### Claim 19 (depends on 14) The computer system of claim 14, wherein the automatic speech recognition system is based on an automatic speech recognition model of the speech data, and wherein the method further comprises: modifying the automatic speech recognition model using independent text data that is different from text data associated with the speech data. ### Claim 20 (depends on 14) The computer system of claim 14, wherein the program code is provided as a service in a cloud environment.