Towards end-to-end speech-input conversational large language models

A system comprising:

• a non-transitory memory including instructions stored thereon;
• and
• a processor operably coupled to the non-transitory memory and configured to execute the instructions of:
• receiving audio from a user;
• generating, via a trained audio encoder based upon the received audio, an audio embedding sequence;
• receiving, via a trained large language model (LLM), the generated audio embedding sequence and a text embedding sequence, wherein the text embedding sequence is located before or after the generated audio embedding sequence;
• producing, via the trained LLM based upon the text embedding sequence, a textual response associated with the audio received from the user;
• and
• causing to display, via a user interface of the user, the produced textual response.

A non-transitory computer-readable medium storing instructions that, when executed, cause:

• receiving audio from a user;
• generating, via a trained audio encoder based upon the received audio, an audio embedding sequence;
• receiving, via a trained large language model (LLM), the generated audio embedding sequence and a text embedding sequence, wherein the text embedding sequence is arranged before or after the generated audio embedding sequence;
• producing, via the trained LLM based upon the text embedding sequence, a textual response associated with the audio received from the user;
• and
• causing to display, via a user interface of the user, the produced textual response.