Extending a parallel file system with a vector database abstraction layer

A method, comprising:

• interfacing, by a device comprising at least one processor, with a metadata server that maintains vector indexes associated with data objects, wherein a vector index of the vector indexes relates to a similarity metric of an associated data object of the data objects used by a vector database;
• using, by the device, a parallel file system comprising a vector database abstraction layer (VDAL) to perform an accelerated similarity query, wherein the VDAL comprises a defined layer within the parallel file system that integrates vector indexing and similarity query coordination with core file system sharding and partitioning operations, wherein the VDAL further comprises vector embeddings used to describe the data objects, and wherein the parallel file system is configured to distribute different portions of the data objects and associated portions of the vector embeddings or vector indexes across multiple nodes or multiple shards of the parallel file system;
• in response to receiving a similarity query that references a vector embedding of the vector embeddings, broadcasting, by the device via the VDAL, the similarity query to the multiple shards in parallel;
• aggregating, by the device via the VDAL, search results from the multiple shards for return to a requesting client;
• and
• executing, by the device via the VDAL, data distribution logic to shard or partition the vector indexes and associated data objects across the multiple shards, enabling higher throughput for ingestion, indexing, or retrieval for vector database operations.

A non-transitory computer-readable medium comprising instructions that, in response to execution, cause a system comprising a processor to perform operations, comprising:

• interfacing with a metadata server that maintains metadata records and vector indexes associated with data objects, wherein a vector index of the vector indexes relates to a similarity metric of an associated data object of the data objects used by a vector database;
• leveraging a parallel file system comprising a vector database abstraction layer (VDAL) to perform an accelerated similarity query,
• wherein the VDAL comprises a defined layer within the parallel file system that integrates vector indexing and similarity query coordination with core file system sharding and partitioning operations,
• wherein the VDAL further comprises vector embeddings used to describe the data objects, and
• wherein the parallel file system is configured to distribute different portions of the data objects and associated portions of the vector embeddings across multiple nodes or multiple shards of the parallel file system;
• in response to receiving a similarity query that references a vector embedding of the vector embeddings, broadcasting, via the VDAL, the similarity query to the multiple shards in parallel;
• aggregating, via the VDAL, search results from the multiple shards for return to a requesting client;
• and
• executing, via the VDAL, data distribution logic to shard or partition the vector indexes and associated data objects across the multiple shards, enabling an increased throughput applicable to ingestion, indexing, or retrieval for vector database operations.