# US 12,119,008 B2 · End-to-end integration of dialog history for spoken language understanding > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,119,008 B2 - Document ID: US12119008B2 - Status: Granted - Kind: B2 - Application number: 17/655,441 - Filed: 2022-03-18 - Published: 2023-09-21 - Granted: 2024-10-15 - Earliest claimed: 2022-03-18 - Assignees: International Business Machines Corporation; The Ohio State University - Applicants: International Business Machines Corporation; The Ohio State University - Inventors: Samuel Thomas; Vishal Sunder; Hong-Kwang Kuo; Jatin Ganhotra; Brian E. D. Kingsbury; Eric Fosler-Lussier - Web page: https://patentgrep.com/patent/US12119008B2 ## CPC - G06F 40/35: Discourse or dialogue representation - G10L 19/00: Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis (in musical instruments G10H) - G06F 40/126: Character encoding - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/096: Transfer learning - G10L 15/00: Speech recognition (G10L17/00 takes precedence) - G10L 15/063: {Training} - G10L 15/16: using artificial neural networks - G10L 15/1822: {Parsing for meaning understanding} - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Systems, computer-implemented methods, and computer program products to facilitate end to end integration of dialogue history for spoken language understanding are provided. According to an embodiment, a system can comprise a processor that executes components stored in memory. The computer executable components comprise a conversation component that encodes speech-based content of an utterance and text-based content of the utterance into a uniform representation. ## Claims ### Claim 1 A system comprising: a memory that stores computer executable components; a processor that executes at least one of the computer executable components that: trains a hierarchical conversational neural network model to generate spoken language understandings directly of speech dialogs in an audio modality without converting the speech dialogs to a text modality, wherein the training comprises: encoding, using a text encoder of the hierarchical conversational neural network model, utterances of a training speech dialog in the audio modality, converted into the text modality, into first embeddings in a uniform embedding representation; encoding, using a speech encoder of the hierarchical conversational neural network model, the utterances of the training speech dialog in the audio modality, without being converted into the text modality, into second embeddings in the uniform embedding representation; and training, using the first embeddings of the utterances in the text modality and the second embeddings of the utterances in the audio modality, a conversation encoder of the hierarchical conversational neural network model to generate a spoken language understanding of the training speech dialog in the audio modality without converting the training speech dialog to the text modality. ### Claim 2 (depends on 1) The system of claim 1, wherein the training of the hierarchical conversational neural network model further comprises: training, using the first embeddings of the utterances in the text modality, the speech encoder, to encode utterances of the speech dialogs in the audio modality into the second embeddings in the uniform embedding representation. ### Claim 3 (depends on 1) The system of claim 1, wherein the training of the hierarchical conversational neural network model further comprises: using one or more cross-model loss functions to transfer semantic knowledge from the text encoder to at least one of the speech encoder or the conversation encoder. ### Claim 4 (depends on 1) The system of claim 1, wherein the at least one of the computer executable components further: generates, using the hierarchical conversational neural network model, a spoken language understanding of at least a portion of a speech dialog in the audio modality without converting the speech dialog to the text modality. ### Claim 5 (depends on 1) The system of claim 1, wherein the speech encoder is a student in a student-teacher joint training framework; and wherein the text encoder is a teacher in the student-teacher joint training framework. ### Claim 6 (depends on 1) The system of claim 1, wherein the training of the hierarchical conversational neural network model further comprises dropping one or more speech frames of the training speech dialog during the training based on a hyperparameter. ### Claim 7 (depends on 3) The system of claim 3, wherein the one or more cross-model loss functions comprise at least one of a Euclidean loss function or a Contrastive loss function. ### Claim 8 A computer-implemented method comprising: training, by a system operatively coupled to a processor, a hierarchical conversational neural network model to generate spoken language understandings directly of speech dialogs in an audio modality without converting the speech dialogs to a text modality, wherein the training comprises: encoding, using a text encoder of the hierarchical conversational neural network model, utterances of a training speech dialog in the audio modality, converted into the text modality, into first embeddings in a uniform embedding representation; encoding, using a speech encoder of the hierarchical conversational neural network model, the utterances of the training speech dialog in the audio modality, without being converted into the text modality, into second embeddings in the uniform embedding representation; and training, using the first embeddings of the utterances in the text modality and the second embeddings of the utterances in the audio modality, a conversation encoder of the hierarchical conversational neural network model to generate a spoken language understanding of the training speech dialog in the audio modality without converting the training speech dialog to the text modality. ### Claim 9 (depends on 8) The computer-implemented method of claim 8, wherein the training of the hierarchical conversational neural network model further comprises: training, using the first embeddings of the utterances in the text modality, the speech encoder, to encode utterances of the speech dialogs in the audio modality into the second embeddings in the uniform embedding representation. ### Claim 10 (depends on 8) The computer-implemented method of claim 8, wherein the training of the hierarchical conversational neural network model further comprises: using one or more cross-model loss functions to transfer semantic knowledge from the text encoder to at least one of the speech encoder or the conversation encoder. ### Claim 11 (depends on 8) The computer-implemented method of claim 8, further comprising: generating, by the system, using the hierarchical conversational neural network model, a spoken language understanding of at least a portion of a speech dialog in the audio modality without converting the speech dialog to the text modality. ### Claim 12 (depends on 8) The computer-implemented method of claim 8, wherein the speech encoder is a student in a student-teacher joint training framework, and the text encoder is a teacher in the student-teacher joint training framework. ### Claim 13 (depends on 8) The computer-implemented method of claim 8, wherein the training of the hierarchical conversational neural network model further comprises: dropping one or more speech frames of the training speech dialog during the training based on a hyperparameter. ### Claim 14 (depends on 10) The computer-implemented method of claim 10, wherein the one or more cross-model loss functions comprise at least one of a Euclidean loss function or a Contrastive loss function. ### Claim 15 A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to: train a hierarchical conversational neural network model to generate spoken language understandings directly of speech dialogs in an audio modality without converting the speech dialogs to a text modality, wherein the training comprises: encoding, using a text encoder of the hierarchical conversational neural network model, utterances of a training speech dialog in the audio modality, converted into the text modality, into first embeddings in a uniform embedding representation; encoding, using a speech encoder of the hierarchical conversational neural network model, the utterances of the training speech dialog in the audio modality, without being converted into the text modality, into second embeddings in the uniform embedding representation; and training, using the first embeddings of the utterances in the text modality and the second embeddings of the utterances in the audio modality, a conversation encoder of the hierarchical conversational neural network model to generate a spoken language understanding of the training speech dialog in the audio modality without converting the training speech dialog to the text modality. ### Claim 16 (depends on 15) The computer program product of claim 15, wherein the training of the hierarchical conversational neural network model further comprises: training, using the first embeddings of the utterances in the text modality, the speech encoder, to encode utterances of the speech dialogs in the audio modality into the second embeddings in the uniform embedding representation. ### Claim 17 (depends on 15) The computer program product of claim 15, wherein the training of the hierarchical conversational neural network model further comprises: using one or more cross-model loss functions to transfer semantic knowledge from the text encoder to at least one of the speech encoder or the conversation encoder. ### Claim 18 (depends on 15) The computer program product of claim 15, wherein the program instructions are further executable by the processor to cause the processor to: generate, using the hierarchical conversational neural network model, a spoken language understanding of at least a portion of a speech dialog in the audio modality without converting the speech dialog to the text modality. ### Claim 19 (depends on 15) The computer program product of claim 15, wherein the training of the hierarchical conversational neural network model further comprises: dropping one or more speech frames are dropped during training based on a hyperparameter. ### Claim 20 (depends on 17) The computer program product of claim 17, wherein the one or more cross-model loss functions comprise at least one of a Euclidean loss function or a Contrastive loss function.