Vector database lookups and/or inference

A method comprising:

• receiving a request to perform vector database lookup for a query;
• performing the vector database lookup by:
• sampling a candidate set of nodes to select a plurality of nodes of the vector database to evaluate,
• for neighbor nodes of the selected plurality of nodes not in memory of a first type, prefetching the neighbors into the memory of the first type from memory of a second type,
• evaluating the neighbors in the memory of the first type by performing a dot product between the selected node(s) and their neighbors irrespective of order in the candidate set of nodes, and
• updating the candidate set in-order based on the dot products;
• and returning the candidate set.

A system comprising:

• compute hardware including at least one accelerator and at least one central processing unit (CPU);
• memory of a first type to store one or more of a machine learning model including at least a proper subset of a key-value (KV) cache and weights, a machine learning model memory manager, and/or a vector database search module;
• and
• memory of a second type to store one or more of at least a proper subset of the KV cache and weights of the machine learning model and/or the vector database, wherein the machine learning model memory manager is to determine and implement a memory split between the memory of the first type and the memory of the second type for the KV cache and weights of the machine learning model and the vector database search module is to perform an opportunistic, out-of-order vector database lookup upon receiving a vector database request.