Improving accuracy of Gen. AI driven document analysis

An Artificial Intelligence (AI) based document analysis method, comprising:

• generating via pre-processing structured and unstructured data sources, a data taxonomy extracted from the structured and unstructured data sources and a custom synonym file that lists domain-specific synonyms extracted from the structured and unstructured data sources;
• selecting, from a plurality of Large Language Models (LLMs) one or more LLMs that can provide response to user queries, wherein the one or more LLMs are selected by an Artificial Intelligence (AI)-based model tester;
• determining that a response to a received user requirement is to be generated;
• retrieving using the data taxonomy, relevant paragraphs for the received user requirement from one or more of the structured and unstructured data sources, wherein relevant documents are retrieved by;
• accessing text data from the structured and unstructured data sources;
• generating paragraph chunks from the text data;
• for each paragraph chunk:
• identify and extract entities and relationships between the entities by using Named Entity Recognition (NER) and one or more key phrase extraction techniques;
• map the identified entities to the data taxonomy;
• seed the data taxonomy with facets and filters corresponding to the identified entities, the facets and filters being represented as nodes and sub-nodes;
• and
• build one or more knowledge graphs representing the identified entities and the relationships between the identified entities;
• generating a prompt to the one or more LLMs from the relevant documents based at least on the received user requirement, wherein the prompt is generated to include less than a predetermined number of tokens arranged in complete sentences;
• obtaining a response to the prompt from each of the one or more LLMs;
• receiving user feedback to a plurality of the responses from the one or more LLMs provided as replies to the received user requirement;
• and
• further training the AI-based model tester on the user feedback for LLM selection.

A non-transitory processor-readable storage medium comprising machine-readable instructions that cause a processor to:

• generate via pre-processing structured and unstructured data sources, a data taxonomy extracted from the structured and unstructured data sources and a custom synonym file that lists domain-specific synonyms extracted from the structured and unstructured data sources;
• select, from a plurality of Large Language Models (LLMs) one or more LLMs that can provide response to user queries, wherein the one or more LLMs are selected by an Artificial Intelligence (AI)-based model tester;
• determine that a response to a received user requirement is to be generated;
• retrieve using the data taxonomy, relevant paragraphs for the received user requirement from one or more of the structured and unstructured data sources, wherein relevant documents are retrieved by:
• accessing text data from the structured and unstructured data sources;
• generating paragraph chunks from the text data;
• for each paragraph chunk:
• identify and extract entities and relationships between the entities by using Named Entity Recognition (NER) and one or more key phrase extraction techniques;
• map the identified entities to the data taxonomy;
• seed the data taxonomy with facets and filters corresponding to the identified entities, the facets and filters being represented as nodes and sub-nodes;
• and
• build one or more knowledge graphs representing the identified entities and the relationships between the identified entities;
• generate a prompt to the one or more LLMs from the relevant documents based at least one the received user requirement, wherein the at least one prompt includes at least a chunk having less than a predetermined number of tokens arranged in complete sentences;
• obtain a response to the prompt from each of the one or more LLMs;
• receive user feedback to the responses obtained from the one or more LLMs provided;
• and
• further train the AI-based model tester on the user feedback for LLM selection.