Artificial intelligence data platform
AssigneeNetApp, Inc.
InventorsArindam Banerjee, Kiran Srinivasan, Leela Shankar Tamma, Atul Ramesh Pandit, Ardalan Kangarlou
Life of this patent
Estimating the term…
Abstract
The disclosure describes artificial intelligence (AI) data platform that utilizes snapshots obtained from a storage node to update a vector database. The AI data platform compares snapshots to generate differential snapshots that identify changed data in storage volumes. The AI data platform uses the differential snapshots to update vector embeddings in a vector database for retrieval-augmented generation (RAG) workflows.
Claims20
A data management system comprising:
- aggregated storage comprising a volume;
- a storage node configured to:
- read and write data to the volume, and
- generate a snapshot of the volume;
- and
- an artificial intelligence (AI) data management node configured to:
- obtain the snapshot from the storage node,
- generate a differential snapshot based on the snapshot and a previous snapshot of the volume, wherein the differential snapshot identifies changed data in the volume, and wherein the identified changed data comprises at least an identification of a new or modified data object in the volume,
- evaluate metadata associated with the new or modified data object to obtain a determination that the new or modified data object satisfies one or more policies, and
- in response to the determination, process the differential snapshot to update a vector database for retrieval augmented generation, wherein updating the vector database comprises generating a vector embedding for the new or modified data object captured in the differential snapshot.
A computer-implemented method comprising, by an artificial intelligence (AI) data management node:
- maintaining a vector database comprising vector embeddings of data stored in one or more storage devices using a volume by a storage node of a storage system;
- generating a differential snapshot based on snapshots obtained from the storage node, wherein the differential snapshot identifies changed data in the volume, and wherein the identified changed data comprises at least an identification of a new or modified data object in the volume;
- evaluating metadata associated with the new or modified data object to obtain a determination that the new or modified data object satisfies one or more policies;
- and
- in response to the determination, vectorizing the changed data to update a vector database for retrieval augmented generation, wherein updating the vector database comprises generating a vector embedding for the new or modified data object captured in the differential snapshot.
A computer-readable storage media device having program instructions stored thereon that, upon execution by one or more processors, cause the one or more processors to:
- maintain a vector database comprising vector embeddings of data in a volume of aggregated storage;
- obtain a differential snapshot from a storage node, wherein the differential snapshot identifies changed data in the volume, the changed data comprising one or more of:
- new data objects or modified data objects;
- evaluate metadata associated with the changed data to identify a set of data objects in the changed data that satisfy one or more policies;
- and
- vectorize the set of identified data objects to update a vector database for retrieval augmented generation.
Dependencies are inferred from the claim text.