System and Method for a Large Codeword Model for Deep Learning

A system for a generic compound large codeword model for natively multimodal deep learning, comprising one or more computers with executable instructions that, when executed, cause the system to:

• receive a plurality of inputs of different modalities;
• convert the plurality of inputs to a unified representation using a modality-agnostic encoder prior to tokenization, wherein the unified representation is a fixed-size tensor in which different sections encode information from different modalities in a common semantic space;
• tokenize the unified representation into a plurality of sourceblocks;
• assign the plurality of sourceblocks a plurality of codewords stored in a codebook library;
• cluster the plurality of codewords based on semantic similarity and learn a single embedding vector for each codeword cluster rather than learning embeddings for individual codewords, thereby reducing the number of embedding vectors processed by the multimodal machine learning core;
• process the plurality of codewords through a multimodal machine learning core comprising an embedding layer and a cross-modal attention mechanism configured to operate on the cluster-level embedding vectors in the shared semantic space;
• generate a codeword response to the plurality of inputs using the multimodal machine learning core;
• translate the codeword response into a translated response which matches one or more modalities of the inputs;
• and
• train the multimodal machine learning core simultaneously on the multiple modalities using a joint loss function that applies uncertainty weighting to automatically balance training contributions from the different modalities within the multimodal machine learning core.