Embeddings-based index for content similarity operations in object stores
AssigneeDell Products L.P.
InventorRaúl Gracia-Tinedo
Life of this patent
Estimating the term…
Abstract
Generating embeddings offline for content similarity functionality is disclosed. Objects stored in a storage system are processed offline to generate embeddings. The embeddings are stored in an embeddings index. The process of generating the embeddings is guided by policies. Content similarity searches may be performed inline by generating embeddings for an input object and then searching the embeddings index based on the input embeddings for the input object. The embeddings index allows additional functionality to be implemented based on the content-similarity search.
Claims20
In a system that includes a storage system that is associated with an embeddings engine, a method comprising:
- sending an event to a write queue associated with an embedding engine configured to perform embeddings operations, wherein the event includes writing an object to a storage of the storage system, and wherein the write queue buffers events such that embeddings processing is decoupled from object write operations;
- processing the event in the write queue by evaluating policies available to the embedding engine to identify a policy applicable to the object, wherein events in the write queue are processed eventually and offline by the embeddings engine;
- retrieving the object and generating embeddings of the object in accordance with the policy, wherein the embeddings represent content of the object and wherein generating the embeddings does not impact read or write operations of the storage system;
- and
- storing the embeddings in an embeddings index, wherein the embeddings index is configured to facilitate content similarity searches.
(canceled)
(canceled)
A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations in a system that includes a storage system associated with an embeddings engine, the operations comprising:
- sending an event to a write queue associated with an embedding engine configured to perform embeddings operations, wherein the event includes writing an object to a storage of the storage system, and wherein the write queue buffers events such that embeddings processing is decoupled from object write operations;
- processing the event by evaluating policies of associated with the embedding engine to identify a policy applicable to the object, wherein events in the write queue are processed eventually and offline by the embeddings engine;
- retrieving the object and generating embeddings of the object in accordance with the policy, wherein the embeddings represent content of the object and wherein generating the embeddings does not impact normal read or write operations of the storage system;
- and
- storing the embeddings in an embeddings index, wherein the embeddings index is configured to facilitate content similarity searches.
(canceled)
Dependencies are inferred from the claim text.