US 10,649,967 B2 · Memory object pool use in a distributed index and query system

Abstract

In a method of memory object pool use in a distributed index and query system, a plurality of memory object pools is maintained. A batch of documents is serialized into a first memory object borrowed from a first memory object pool, of the memory object pools, to achieve a first segment comprising a serialized data array, an offset index, and a document count. The serialized data array is compressed into a second memory object borrowed from a second memory object pool, of the memory object pools, to achieve a second segment comprising a compressed serialized data array, the offset index, and the document count. Subsequent to the compression, the first memory object is released back to the first memory object pool. The second segment is written to a data storage device. Subsequent to the writing, the second memory object is released back to the second memory object pool.

Claims20

Dependent claims

A method of memory object pool use in a distributed index and query system, the method comprising:

maintaining, in a memory of the distributed index and query system, a plurality of memory object pools;
serializing a batch of documents into a first memory object borrowed from a first memory object pool of the plurality of memory object pools to achieve a first segment comprising a serialized data array, an offset index describing locations of documents of the batch of documents within the serialized data array, and a document count describing a total number of the documents;
compressing the serialized data array into a second memory object borrowed from a second memory object pool of the plurality of memory object pools to achieve a second segment comprising a compressed serialized data array, the offset index, and the document count;
subsequent to the compressing, releasing the first memory object back to the first memory object pool;
writing the second segment to a data storage device;
and
subsequent to the writing, releasing the second memory object back to the second memory object pool.

4 more elements

A non-transitory computer readable storage medium having computer readable program code stored thereon for causing a computer system to perform a method of memory object pool use in a distributed index and query system, the method comprising:

maintaining, in a memory of the distributed index and query system by a memory manager of the distributed index and query system, a plurality of memory object pools;
in response to a query, reading a first segment from a data storage device into a first memory object borrowed from a first memory object pool of the plurality of memory object pools, the first segment comprising a compressed serialized data array, an offset index describing locations of documents in an uncompressed version of the compressed serialized data array, and a document count describing a number of the documents in the compressed serialized data array;
decompressing the compressed serialized data array into a second memory object borrowed from a second memory object pool of the plurality of memory object pools to achieve a second segment comprising the uncompressed version of the serialized data array, the offset index, and the document count;
subsequent to the decompressing, releasing the first memory object back to the first memory object pool;
and
de-serializing a document from the second memory object to satisfy the query.

3 more elements

A distributed index and query system comprising:

a data storage device;
a plurality of processors, the plurality of processors comprising a plurality of processing threads;
a memory coupled with the data storage device and the plurality of processors, the memory comprising a plurality of memory object pools;
and
a memory manager configured to maintain the plurality of memory object pools;
and
wherein a first processing thread of the plurality of processing threads is configured to:
serialize a batch of documents into a first memory object borrowed from a first memory object pool of the plurality of memory object pools to achieve a first segment comprising a serialized data array, an offset index describing locations of documents of the batch of documents within the serialized data array, and a document count describing a total number of the documents;
compress the serialized data array into a second memory object borrowed from a second memory object pool of the plurality of memory object pools to achieve a second segment comprising a compressed serialized data array, the offset index, and the document count;
subsequent to the compression, release the first memory object back to the first memory object pool;
write the second segment to a data storage device;
and
subsequent to the writing, release the second memory object back to the second memory object pool.

10 more elements

Dependencies are inferred from the claim text.

Memory object pool use in a distributed index and query system

Life of this patent

Memory object pool use in a distributed index and query system

Life of this patent

Abstract

Claims20