US 12,596,689 B2Granted

Dynamic storage journaling partitions for efficient resource use and improved system throughput

AssigneeNetApp, Inc.

InventorsParag Sarfare, Asif Pathan, Amit Borase, Nikhil Mattankot, Sudheer Kumar Vavilapalli, Stephen Wu

Life of this patent

Estimating the term…

Loading patent…

Abstract

Systems and methods are for improving latency and throughput for metadata-heavy workloads and/or workloads including metadata bursts by decoupling data journal records and metadata-only journal records are provided. According to one embodiment, expedited and independent space reclamation is facilitated by differentiating between various types of journal records chains of which should be retained until different conditions are met. For example, data journal records may be added to data journal record chains within a persistent KV store and metadata-only journal records may be added to metadata-only journal record chains within the persistent KV store. Reclamation of spaced utilized by a data journal record chain may be reclaimed after both remote node data flush has been completed and the completion of a local CP for all records in the chain, whereas records of a metadata-only journal chain may be freed independently upon completion of a local CP for all records.

Claims20

Dependent claims

A method comprising:

maintaining, within a cluster of a plurality of nodes representing a distributed storage system, an operation log of file system operations that have yet to be flushed to storage of the distributed storage system, including (i) decoupling a plurality of types of journal records from each other by organizing the operation log as a plurality of chains of journal records in which all journal records of a given chain of the plurality of chains of journal records are of a same type of the plurality of types of journal records and (ii) storing the operation log within a persistent key-value (KV) store, wherein the plurality of types of journal records include (a) a data journal record having both data and metadata of a given file system operation at issue and (b) a metadata-only journal record having metadata of the given file system operation at issue;
receiving a storage request by a node of the plurality of nodes;
dynamically determining, by the node, based on an operation associated with the storage request, a type of journal record to be used to log the operation within the persistent KV store from among the plurality of types of journal records;
based on the determined type of journal record being the metadata-only journal record, logging, by the node, the operation and corresponding metadata within a new or an active chain of metadata-only journal records within the persistent KV store;
based on the determined type of journal record being the data journal record, logging, by the node, the operation, corresponding data, and the corresponding metadata within a new or an active chain of data journal records within the persistent KV store;
and
based on their respective different sets of one or more reclamation dependencies, independently reclaiming space in the persistent KV store associated with a plurality of metadata-only journal records and a plurality of data journal records of respective frozen chains of a plurality of frozen chains within the persistent KV store.

4 more elements

A non-transitory machine readable medium storing instructions, which when executed by one or more processing resources of a node of a plurality of nodes of a cluster representing a distributed storage system, cause the node to:

maintain an operation log of file system operations that have yet to be flushed to storage of the distributed storage system, including (i) decoupling a plurality of types of journal records from each other by organizing the operation log as a plurality of chains of journal records in which all journal records of a given chain of the plurality of chains of journal records are of a same type of the plurality of types of journal records and (ii) storing the operation log within a persistent key-value (KV) store, wherein the plurality of types of journal records include (a) a data journal record having both data and metadata of a given file system operation at issue and (b) a metadata-only journal record having metadata of the given file system operation at issue;
receive a storage request from a client of the distributed storage system or a file system of the node;
determine a type of journal record to be used to log an operation associated with the storage request within the persistent KV store from among the plurality of types of journal records;
based on determining the type of journal record to be used is the metadata-only journal record, log the operation and corresponding metadata within a new or an active chain of metadata-only journal records within the persistent KV store;
and
based on determining the type of journal record to be used is the data journal record, log the operation, corresponding data, and the corresponding metadata within a new or an active chain of data journal records within the persistent KV store;
and
based on their respective different sets of one or more reclamation dependencies, independently reclaim space in the persistent KV store associated with a plurality of metadata-only journal records and a plurality of data journal records of respective frozen chains of a plurality of frozen chains within the persistent KV store.

5 more elements

A node of a plurality of nodes of a cluster of a distributed storage system comprising:

one or more processing resources;
and
instructions that when executed by the one or more processing resources cause the node to:
maintain an operation log of file system operations that have yet to be flushed to storage of the distributed storage system, including (i) decoupling a plurality of types of journal records from each other by organizing the operation log as a plurality of chains of journal records in which all journal records of a given chain of the plurality of chains of journal records are of a same type of the plurality of types of journal records and (ii) storing the operation log within a persistent key-value (KV) store, wherein the plurality of types of journal records include (a) a data journal record having both data and metadata of a given file system operation at issue and (b) a metadata-only journal record having metadata of the given file system operation at issue;
receive a storage request from a client of the distributed storage system or a file system operable within the node;
determine a type of journal record to be used to log an operation associated with the storage request within the persistent KV store from among the plurality of types of journal records;
based on determining the type of journal record to be used is the metadata-only journal record, log the operation and corresponding metadata within a new or an active chain of metadata-only journal records within the persistent KV store;
and
based on determining the type of journal record to be used is the data journal record, log the operation, corresponding data, and the corresponding metadata within a new or an active chain of data journal records within the persistent KV store;
and
based on their respective different sets of one or more reclamation dependencies, independently reclaim space in the persistent KV store associated with a plurality of metadata-only journal records and a plurality of data journal records of respective frozen chains of a plurality of frozen chains within the persistent KV store.

8 more elements

Dependencies are inferred from the claim text.