Systems and methods for an indexing tree based artificial intelligence conversation agent

A system for retrieval-augmented text generation to generate a response to a user query by an artificial intelligence (AI) agent, the system comprising:

• a memory that stores one or more neural network based language models and a plurality of processor executable instructions;
• a communication interface that receives the user query;
• and
• one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory to perform operations comprising:
• constructing, by one or more neural network based language models, an indexing tree reflecting sematic relationship among textual contexts of one or more documents, wherein the indexing tree comprises at least a first level of nodes representing a plurality of propositions indicating facts about at least one entity from the one or more documents, a second level of nodes representing a plurality of proposition aggregates that aggregate one or more first-level propositions mentioning a same entity, and a third level of nodes representing a plurality of summaries that summarize one or more second-level proposition aggregates;
• receiving, via a communication interface, the user query;
• retrieving, from the indexing tree, one or more nodes across multiple levels based on similarity comparison with the user query based at least on a similarity between embeddings of the one or more nodes and an embedding of the user query;
• generating, by the neural network based language model, a response in response to the user query based on a context combining at least summaries and/or proposition aggregates corresponding to the retrieved one or more nodes;
• and
• causing the response to be displayed at a visualized user interface of the AI agent.

A non-transitory machine-readable medium comprising a plurality of machine-executable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising:

• constructing, by one or more neural network based language models, an indexing tree reflecting sematic relationship among textual contexts of one or more documents, wherein the indexing tree comprises at least a first level of nodes representing a plurality of propositions indicating facts about at least one entity from the one or more documents, a second level of nodes representing a plurality of proposition aggregates that aggregate one or more first-level propositions mentioning a same entity, and a third level of nodes representing a plurality of summaries that summarize one or more second-level proposition aggregates;
• receiving, via a communication interface, the user query;
• retrieving, from the indexing tree, one or more nodes across multiple levels based on similarity comparison with the user query based at least on a similarity between embeddings of the one or more nodes and an embedding of the user query;
• generating, by the neural network based language model, a response in response to the user query based on a context combining at least summaries and/or proposition aggregates corresponding to the retrieved one or more nodes;
• and
• causing the response to be displayed at a visualized user interface of the AI agent.