Memory allocation buffer for reduction of heap fragmentation
AssigneeCloudera, Inc.
InventorTodd Lipcon
Life of this patent
Estimating the term…
Abstract
Systems and methods of a memory allocation buffer to reduce heap fragmentation. In one embodiment, the memory allocation buffer structures a memory arena dedicated to a target region that is one of a plurality of regions in a server in a database cluster such as an HBase cluster. The memory area has a chunk size (e.g., 2 MB) and an offset pointer. Data objects in write requests targeted to the region are received and inserted to the memory arena at a location specified by the offset pointer. When the memory arena is filled, a new one is allocated. When a MemStore of the target region is flushed, the entire memory arenas for the target region are freed up. This reduces heap fragmentation that is responsible for long and/or frequent garbage collection pauses.
Claims24
A method for operating a region server of a distributed computing system, the method comprising:
- tracking usage of a memory arena in a memory heap of the region server, the memory arena dedicated to a target region of the distributed computing system, wherein the memory arena is configured to utilize an offset pointer to track a previous end location associated with a previous write request so that a subsequent write request can continue data writing from the previous end location, and wherein the offset pointer is reset upon allocating a new chunk of blocks for the target region;
- determining, based on the tracking, that usage of the memory arena has reached a usage threshold;
- and
- flushing data from the memory arena to a storage in response to determining that usage of the memory arena has reached the usage threshold;
- wherein the memory arena is a contiguous portion of the memory heap such that when said flushing occurs data storage space freed from the memory arena is also contiguous.
A server hosting a plurality of regions for a distributed computing system, the server including a memory storing instructions which, when executed by one or more processors of the server, cause the server to perform operations comprising:
- tracking usage of a memory arena in a memory heap of the server, the memory arena dedicated to a target region of the plurality of regions, wherein the memory arena is configured to utilize an offset pointer to track a previous end location associated with a previous write request so that a subsequent write request can continue data writing from the previous end location, and wherein the offset pointer is reset upon allocating a new chunk of blocks for the target region;
- determining, based on the tracking, that usage of the memory arena has reached a usage threshold;
- and
- flushing data from the memory arena to a storage in response to determining that usage of the memory arena has reached the usage threshold;
- wherein the memory arena is a contiguous portion of the memory heap such that when said flushing occurs data storage space freed from the memory arena is also contiguous.
A non-transitory machine-readable storage medium including instructions which, when executed by one or more processors of a server hosting a plurality of regions for a distributed computing system, cause the server to perform operations comprising:
- tracking usage of a memory arena in a memory heap of the server, the memory arena dedicated to a target region of the plurality of regions, wherein the memory arena is configured to utilize an offset pointer to track a previous end location associated with a previous write request so that a subsequent write request can continue data writing from the previous end location, and wherein the offset pointer is reset upon allocating a new chunk of blocks for the target region;
- determining, based on the tracking, that usage of the memory arena has reached a usage threshold;
- and
- flushing data from the memory arena to a storage in response to determining that usage of the memory arena has reached the usage threshold;
- wherein the memory arena is a contiguous portion of the memory heap such that when said flushing occurs space freed from the memory arena is also contiguous.
Dependencies are inferred from the claim text.