Multi-tenancy retrieval-access generation ingestion versioning

A method, comprising:

• storing, by a system comprising at least one processor, a group of checkpoints that comprise pairs, respective pairs of the pairs comprising respective identifications of respective data stored in a storage system and respective second generation identifiers that correspond to the respective data;
• and
• based on executing a retrieval-augmented generation process comprising performance of an iteration of ingesting the data from the storage system and to send the data to be ingested by a first retrieval-augmented generation system of a group of retrieval-augmented generation systems,
• identifying, by the system, a first checkpoint among the group of checkpoints based on the first checkpoint comprising an indication of the first retrieval-augmented generation system,
• querying, by the system, a search system to identify a first portion of the data having respective third generation identifiers that are greater than the respective second generation identifiers in the first checkpoint, wherein the search system stores respective metadata of the at least some respective data from the storage system, and wherein the respective metadata comprises respective first generation identifiers that indicate respective updates to the respective data,
• ingesting, by the system, the first portion of the data into the first retrieval-augmented generation system while refraining from ingesting a second portion of the data having corresponding fourth generation identifiers that are less than or equal to the corresponding second generation identifiers in the first checkpoint, and
• servicing, by the system, queries to the first retrieval-augmented generation system based on the ingesting of the first portion of the data.

A non-transitory computer-readable medium comprising instructions that, in response to execution, cause a system comprising at least one processor to perform operations, comprising:

• based on executing a retrieval-augmented generation framework comprising performance of an iteration of ingesting data from a storage system and to send the data to be ingested by a first retrieval-augmented generation system, identifying a first state file among a group of state files based on the first state file comprising an indication of the first retrieval-augmented generation system;
• querying a search system to identify a first portion of the data having respective third generation identifiers that are greater than the respective second generation identifiers in the first state file, wherein the search system stores respective metadata of the respective data from the storage system, wherein the respective metadata comprises respective first generation identifiers that indicate respective updates to the respective data;
• ingesting the first portion of the data into the first retrieval-augmented generation system while refraining from ingesting a second portion of the data having corresponding fourth generation identifiers that are less than or equal to the corresponding second generation identifiers in the first state file;
• and
• servicing queries to the first retrieval-augmented generation system based on the ingesting of the first portion of the data.