Speech-text alignment engine in an artificial intelligence system

One or more computer-storage media having computer-executable instructions embodied thereon that, when executed by a computing system having a processor and memory, cause the processor to perform operations, the operations comprising:

• communicating speech data associated with a speech-based task for a client device;
• based on communicating the speech data, receiving a response associated with the speech data and the speech-based task, wherein the response is associated with a speech-text alignment engine that generates aligned embeddings of the speech data based on length matching and latent space matching of the speech data to text data, wherein the response is generated using a Large Language Model (LLM);
• and
• generating the response to the speech-based task at the client device.

A computer-implemented method, the method comprising:

• accessing Automatic Speech Recognition (ASR) training data, the ASR training data comprising speech data and text data;
• using a Connectionist Temporal Classification (CTC) layer, generating alignment information associated with a non-blank token sequence of the speech data and a speech input sequence of the speech data;
• based on the alignment information and the speech data, using a modified q-former layer to generate aligned embeddings;
• using the aligned embeddings, training a Large Language Model (LLM);
• and
• deploying the LLM.