Audio and video tokenization for multimodal large language models

Abstract

Systems and methods for power-efficient, continuous tokenization and long-context storage of audio and video data for use with multimodal large language models (LLMs). The systems include specialized subsystems configured to receive input signals, generate discrete tokens representing the input, and buffer the tokens for durations ranging from seconds to hours. Upon receiving a trigger to initiate communication with a multimodal LLM, at least a subset of the buffered tokens is transmitted to an inference dispatcher, which determines the distribution of the tokens to one or more inference engines for processing. The architecture supports tokenization and buffering for multiple modalities, including audio, video, image, and text, and enables context-rich, privacy-preserving, and low-latency AI interactions on client devices. By utilizing efficient token-based data encoding and performing the tokenization at low-power hardware, power consumption and bandwidth usage are significantly reduced, thereby allowing seamless, always-on multimodal AI experiences on battery-powered platforms.

Dependencies are inferred from the claim text.