Llmvox: autoregressive streaming text-to-speech model for any large language model (llm)

A text-to-speech system for converting an input text generated by a large language model (LLM) into an audio speech signal, the system comprising:

• first and second first-in-first-out (FIFO) buffers;
• a decoder-only transformer;
• processing circuitry configured to:
• receive the input text from the large language model (LLM),
• tokenize the input text into a sequence of phoneme-level tokens,
• assign, based on sentence boundaries detected within the input text, the sequence of phoneme-level tokens to a plurality of adaptive text chunks,
• alternately enqueue the plurality of adaptive text chunks in the first and second first-in-first-out (FIFO) buffers,
• concurrently process the plurality of adaptive text chunks in the first and second FIFO buffers, wherein the concurrently processing includes, for each adaptive text chunk in a respective one of the first and second FIFO buffers:
• map each phoneme-level token in the adaptive text chunk to a phoneme embedding vector,
• concatenate each phoneme embedding vector with a latent acoustic feature vector derived from a previous speech token or initialized as a zero tensor to form a combined feature vector,
• add a learnable positional embedding vector to each combined feature vector to form a sequence of position-embedded feature vectors, and
• feed the sequence of position-embedded feature vectors to the decoder-only transformer to cause the decoder-only transformer to autoregressively predict subsequent speech tokens for the adaptive text chunk in a streaming manner;
• and
• a neural audio codec configured to decode the predicted speech tokens generated in the streaming manner to generate a final speech waveform as the audio speech signal.