Dynamic arrangement of data chunks in storage

A computer program product comprising:

• one or more computer-readable storage media;
• and
• program instructions stored on the one or more storage media to perform operations comprising:
• storing pointers in a vector database, the pointers referencing respective source data chunks on a storage device;
• assigning, using a large language model, respective contextual scores to the respective source data chunks based on evaluating data included therein;
• identifying respective source data chunks, for which a difference between respective contextual scores is in a predetermined range, as similar source data chunks;
• rearranging at least some of the respective source data chunks such that the similar source data chunks are logically and/or physically positioned adjacent to each other in the storage device;
• in response to executing a similarity search, receiving a pointer from the vector database;
• using the received pointer to return a respective one of the source data chunks from the storage device;
• and transparently providing the returned source data chunk to an upstream application, wherein the returned source data chunk is provided to the upstream application without the upstream application noticing that the returned source data chunk was returned from external storage rather than being co-located with vector database information, wherein similar ones of the respective source data chunks are co-located in the storage device.

A computer system comprising:

• a processor set;
• one or more computer-readable storage media;
• and
• program instructions stored on the one or more storage media to cause the processor set to perform operations comprising:
• storing pointers in a vector database, the pointers referencing respective source data chunks on a storage device;
• assigning, using a large language model, respective contextual scores to the respective source data chunks based on evaluating data included therein;
• identifying the respective source data chunks, for which a difference between respective contextual scores is in a predetermined range, as similar source data chunks;
• rearranging at least some of the respective source data chunks such that the similar source data chunks are logically and/or physically positioned adjacent to each other in the storage device;
• in response to executing a similarity search, receiving a pointer from the vector database;
• using the received pointer to return a respective one of the source data chunks from the storage device;
• and
• transparently providing the returned source data chunk to an upstream application, wherein the returned source data chunk is provided to the upstream application without the upstream application noticing that the returned source data chunk was returned from external storage rather than being co-located with vector database information, wherein similar ones of the respective source data chunks are co-located in the storage device.