Metering data in distributed storage environments
AssigneeAmazon Technologies, Inc.
InventorsMichael Anthony Nuss, Matti Juhani Oikarinen
Life of this patent
Estimating the term…
Abstract
Techniques described and suggested include systems, methods, and processes for determining data-related attributes, such as size, for data and/or data containers (such as volumes and file systems) resident on distributed storage systems. For example, a shared records storage device may be utilized to gather information related to file systems stored on a distributed storage system having a plurality of storage servers. A data processor may be used to generate information records and meter data storage size for file systems resident on the distributed storage system, and such records may be provided to requestors, such as customers associated with the file systems.
Claims20
A computer-implemented method, comprising:
- under the control of one or more computer systems configured with executable instructions,
- determining a manifest of one or more extents replicated on a plurality of storage servers, each extent of the one or more extents allocating data storage space of the plurality of storage servers, the data storage space being capable of storing the data;
- for each extent of the one or more extents, determining file system information relating to file systems, at least portions of which are stored within the extent;
- processing the determined file system information by at least:
- sorting the file system information such that it is keyed, for each file system for which a portion is stored within at least one extent of the one or more extents in the manifest, to a plurality of file system identifiers within the information;
- and
- updating, for each of the plurality of file system identifiers, at least a subset of file system records by combining the sorted file system information for each of the one or more extents in which a respective portion of a respective file system is stored, such that the updated subset of file system records includes at least a size of the file system identified by the file system identifier;
- and
- for each extent of the one or more extents in the manifest, updating the file systems for which the portions are stored within the extent to include at least a subset of the processed file system information, the subset of the processed file system information including at least the size of each of the file systems.
A system, comprising:
- a plurality of storage devices;
- one or more processors;
- memory including instructions that, when executed by the one or more processors, cause the system to:
- determine one or more extent attributes for a plurality of extents allocated on at least a subset of the plurality of storage devices, the attributes including file system information for each file system within each extent of the plurality of extents;
- process the determined extent attributes for the plurality of storage devices to at least:
- sort the determined extent attributes based on the file system information;
- and
- determine size of each of the file systems by combining the sorted extent attributes with at least one set of prior records relating to the file systems.
A non-transitory computer-readable storage medium having stored thereon executable instructions that, when executed by one or more processors of a computer system, cause the computer system to at least:
- generate, for at least two extents of a plurality of extents allocated on a plurality of storage devices, a first set of records for file system attributes relating to each of a plurality of file systems stored on the extents, the file system attributes including identifiers for each of the plurality of file systems;
- identify duplicate records from the first set of records;
- map the identified duplicate set of records to generate a second set of records that is sorted by the identifiers;
- combine the second set of records with at least one third set of historical records relating to each of the plurality of file systems to determine the size of each of the plurality of file systems, thereby generating a fourth set of records;
- and
- provide programmatic access to the fourth set of records.
Dependencies are inferred from the claim text.