Method and system for optimizing use of retrieval augmented generation pipelines in generative artificial intelligence applications

A system for processing domain-specific content in a generative artificial intelligence system, comprising:

• a processor;
• a network communication device operably coupled to the processor and configured to transmit and receive digital transmissions across a network to removed computerized devices;
• and
• a non-transitory computer-readable storage medium having stored thereon software that, when executed by the processor, is configured to:
• receive a prompt from a user;
• generate a tokenized prompt by performing an initial tokenization of the prompt;
• identify an identified domain comprised by a plurality of supported domains that is associated with the tokenized prompt by analyzing the tokenized prompt;
• identify one or more domain-specific functions within the identified domain by performing a domain-specific analysis on the tokenized prompt;
• generate a plurality of domain-specific sub-functions from the one or more domain-specific functions according to a hierarchical mapping of the identified domain;
• generate a plurality of hierarchical tokens (H-Tokens), each H-Token of the plurality of H-Tokens being configured to encapsulate one of a domain-specific function of the one or more domain-specific functions or a domain-specific sub-function of the plurality of domain-specific sub-functions, and relationships between the one or more domain-specific functions and the plurality of domain-specific sub-functions;
• implement the plurality of H-Tokens, resulting in an implemented plurality of H-Tokens, by mapping each H-Token of the plurality of H-Tokens to each of a corresponding function, event, and implementation method within the identified domain;
• assemble a response using the implemented plurality of H-Tokens;
• and
• transmit the response to the user.

A system for processing domain-specific content in a generative artificial intelligence system, comprising:

• means for receiving a prompt from a user;
• means for generating a tokenized prompt by performing an initial tokenization of the prompt;
• means for identifying an identified domain comprised by a plurality of supported domains that is associated with the tokenized prompt by analyzing the tokenized prompt;
• means for identifying one or more domain-specific functions within the identified domain by performing a domain-specific analysis on the tokenized prompt;
• means for generating a plurality of domain-specific sub-functions from the one or more domain-specific functions according to a hierarchical mapping of the identified domain;
• means for generating a plurality of hierarchical tokens (H-Tokens), each H-Token of the plurality of H-Tokens being configured to encapsulate one of a domain-specific function of the one or more domain-specific functions or a domain-specific sub-function of the plurality of domain-specific sub-functions, and relationships between the one or more domain-specific functions and the plurality of domain-specific sub-functions;
• means for implementing the plurality of H-Tokens, resulting in an implemented plurality of H-Tokens, by mapping each H-Token of the plurality of H-Tokens to each of a corresponding function, event, and implementation method within the identified domain;
• means for assembling a response using the implemented plurality of H-Tokens;
• and
• means for transmitting the response to the user.