# US 2016/0259834 A1 · Usage and bandwidth utilization collection mechanism for a distributed storage system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0259834 A1
- Document ID: US20160259834A1
- Status: Published application
- Kind: A1
- Application number: 14/637,284
- Filed: 2015-03-03
- Published: 2016-09-08
- Earliest claimed: 2015-03-03
- Assignees: SwiftStack, Inc.
- Applicants: SwiftStack, Inc.
- Inventors: Darrell Bishop
- Web page: https://patentgrep.com/patent/US20160259834A1

## CPC

- G06F 11/3034: {where the computing system component is a storage system, e.g. DASD based or network based (digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097)}
- G06F 11/3006: {where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems (multiprogramming arrangements G06F9/46; allocation of resources G06F9/50)}
- G06F 11/3452: {Performance evaluation by statistical analysis}
- G06F 11/3476: {Data logging (G06F11/14, G06F11/2205 take precedence)}
- G06F 16/148: {File search processing}
- G06F 16/182: Distributed file systems
- G06F 16/27: Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- H04L 12/14: Charging, metering or billing arrangements specially adapted for data communications, e.g. authentication, authorisation and accounting [AAA] framework
- H04L 43/0876: Network utilisation, e.g. volume of load or congestion level
- G06F 11/1471: involving logging of persistent data for recovery
- G06F 11/2094: {Redundant storage or storage space (G06F11/2056 takes precedence)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A technique is introduced that enables one or more mechanisms to collect storage and transfer utilization metrics for an account in a distributed data storage system that are more scalable and robust than conventional ways. The technique includes a method comprising, for each of the nodes, of collecting storage utilization data for a one account. The method further includes, for each proxy server, collecting transfer utilization data for the one account. The method further includes, at the controller, generating a cluster account interval (CAI) data based on a raw storage data file and an aggregated transfer data file. The CAI data include storage and transfer utilization data over a predetermined time span.

## Claims

### Claim 1

A method for collecting utilization data for a plurality of accounts in a distributed storage cluster including a controller and a plurality of nodes, each node coupled to a plurality of drives, the method comprising:
for each of the nodes, collecting storage utilization data for a one account by:
extracting, at a given node, a plurality of storage utilization parameters recorded under the one account from each of a plurality of account databases that are maintained by a plurality of drives coupled to the given node until all drives coupled to the given node are processed;
generating a raw storage data file based on the plurality of storage utilization parameters extracted at the given node to the raw storage data file;
storing the raw storage data file on the given node;
locating, by the given node periodically searching for, the raw storage data file on the given node;
upon the raw storage data file being located:
copying the raw storage data file to the cluster until the raw storage data file is successfully copied to the cluster;
upon successfully copying the raw storage data file to the cluster, marking the raw storage data file as copied;
uploading the raw storage data file to the controller until the raw storage data file is successfully uploaded to the controller; and
upon successfully uploading the raw storage data file to the controller, deleting the raw storage data file.

### Claim 2 (depends on 1)

The method of claim 1, wherein the distributed storage cluster further includes a plurality of proxy servers, each proxy server coupled to one or more nodes to coordinate access requests with its corresponding nodes, the method further comprising:
for each proxy server, collecting transfer utilization data for the one account by:
creating, at a given proxy server, a rotated log file based on a log file being maintained by the given proxy server, wherein the log file contains entries for each access request processed by the given proxy server;
locating, by the given proxy server periodically searching for, the rotated log file on the given proxy server;
upon the rotated log file being located:
renaming the rotated log file such that the rotated log file becomes a preprocess file;
parsing the preprocess file on the given proxy server to extract raw transfer data, wherein the raw transfer data parsed from the preprocess file are aggregated into aggregated transfer data until the parsing completes;
transferring, from the given proxy server, the raw transfer data to the cluster;
generating, at the given proxy server, an aggregated transfer data file based on the aggregated transfer data;
determining whether both transferring the raw data and generating the aggregated transfer data file are successful;
if both transferring the raw data and generating the aggregated transfer data file are successful, uploading the aggregated transfer data file to the controller until the aggregated transfer data file is successfully uploaded to the controller; and
upon successfully uploading the aggregated transfer data file to the controller, deleting the aggregated transfer data file.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
at the controller, generating a cluster account interval (CAI) data for the one account based on the raw storage data file and the aggregated transfer data file,
wherein each CAI datum is derived from one record from each of the raw storage data file and the aggregated transfer data file, and
wherein the CAI data include storage and transfer utilization data over a predetermined time span.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
at the controller, generating a cluster account interval offset (CAIO) data for the one account based on the raw storage data file and the aggregated transfer data file,
wherein the CAIO data has the same predetermined time span as the CAI data but has a different starting time than the CAI data.

### Claim 5 (depends on 4)

The method of claim 4, wherein the predetermined time span is one (1) hour, and wherein the starting times for the CAI and CAIO data are offset by half (½) hour.

### Claim 6 (depends on 4)

The method of claim 4, wherein each record from the raw storage data file and the aggregated transfer data file only contributes to exactly one CAI datum and one CAIO datum.

### Claim 7 (depends on 4)

The method of claim 4, further comprising:
at the controller, tracking (1) an expected count of records in the raw storage data file and the aggregated transfer data file, and (2) an actual processed count of records in the raw storage data file and the aggregated transfer data file.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
generating a table tracking a percent complete value based on the tracking.

### Claim 9 (depends on 2)

The method of claim 2, further comprising:
upon successfully receiving the raw storage data file or the aggregated transfer data file, transmitting, by the controller, an acknowledgement for successful receipt.

### Claim 10 (depends on 2)

The method of claim 2, further comprising:
at the cluster, storing the raw transfer data transferred from the given proxy server as a backup file.

### Claim 11 (depends on 2)

The method of claim 2, wherein the renaming comprises appending a unique suffix to a file name of the rotated log file to create the preprocess file.

### Claim 12 (depends on 2)

The method of claim 2, wherein the all available preprocess files have the same prefix in their file names.

### Claim 13 (depends on 2)

The method of claim 2, wherein transferring the raw transfer data and parsing the preprocess file on the given proxy server are performed simultaneously.

### Claim 14 (depends on 2)

The method of claim 2, wherein parsing the preprocess file on the given proxy server is performed on a per-entry basis.

### Claim 15 (depends on 2)

The method of claim 2, wherein the raw transfer data include, for the one account, (1) bytes transferred in from the cluster, and (2) bytes transferred out to the cluster.

### Claim 16 (depends on 15)

The method of claim 15, wherein an in-memory aggregation data structure is maintained on a memory of the given proxy server to store the raw transfer data, the method further comprising:
maintaining the in-memory aggregation data structure for the one account by, for a given entry of the raw transfer data:
incrementing a value representing bytes transferred in from the cluster by a first corresponding value parsed out from the given entry;
incrementing a value representing bytes transferred out to the cluster by a second corresponding value parsed out from the given entry; and
incrementing a value representing request count by one.

### Claim 17 (depends on 2)

The method of claim 2, wherein each entry in the raw transfer data represents one access request.

### Claim 18 (depends on 2)

The method of claim 2, wherein internal proxy sub-requests and unparsable lines in the preprocess files are ignored.

### Claim 19 (depends on 1)

The method of claim 1, wherein the storage utilization parameters include a container count, an object count, and bytes used values for the one account.

### Claim 20

A method for collecting utilization data for a plurality of accounts in a distributed object storage cluster including a controller, a plurality of nodes, each node coupled to a plurality of drives, and a plurality of proxy servers, each proxy server coupled to one or nodes to coordinate access requests with its corresponding nodes, the method comprising:
(1) for each of the nodes, collecting storage utilization data for a one account by:
extracting, at a given node, a plurality of storage utilization parameters recorded under the one account from each of a plurality of account databases that are maintained by a plurality of drives coupled to the given node until all drives coupled to the given node are processed;
generating a raw storage data file based on the plurality of storage utilization parameters extracted at the given node to the raw storage data file;
storing the raw storage data file on the given node;
locating, by periodically searching for, the raw storage data file on the given node; and
upon the raw storage data file being located, moving the raw storage data file to the controller;
(2) for each proxy server, collecting transfer utilization data for the one account by:
creating, at a given proxy server, a rotated log file based on a log file being maintained by the given proxy server, wherein the log file contains entries for each access request processed by the given proxy server;
locating, by periodically searching for, the rotated log file on the given proxy server; and
upon the rotated log file being located, generating and moving an aggregated transfer data file to the controller, wherein the aggregated transfer data file is generated based on the rotated log file; and
(3) at the controller, generating a cluster account interval (CAI) data based on the raw storage data file and the aggregated transfer data file, wherein the CAI data include storage and transfer utilization data over a predetermined time span.

### Claim 21 (depends on 20)

The method of claim 20, wherein moving the raw storage data file to the controller comprises:
copying the raw storage data file to the cluster until the raw storage data file is successfully copied to the cluster;
upon successfully copying the raw storage data file to the cluster, marking the raw storage data file as copied;
uploading the raw storage data file to the controller until the raw storage data file is successfully uploaded to the controller; and
upon successfully uploading the raw storage data file to the controller, deleting the raw storage data file.

### Claim 22

The method of 20, wherein generating and moving the aggregated transfer data file to the controller comprises:
renaming the rotated log file such that the rotated log file becomes a preprocess file;
parsing the preprocess file on the given proxy server to extract raw transfer data, wherein the raw transfer data parsed from the preprocess file are aggregated into aggregated transfer data until the parsing completes;
transferring, from the given proxy server, the raw transfer data to the cluster;
generating, at the given proxy server, an aggregated transfer data file based on the aggregated transfer data;
determining whether both transferring the raw data and generating the aggregated transfer data file are successful;
if both transferring the raw data and generating the aggregated transfer data file are successful, uploading the aggregated transfer data file to the controller until the aggregated transfer data file is successfully uploaded to the controller;
upon successfully uploading the aggregated transfer data file to the controller, deleting the aggregated transfer data file.

### Claim 23

A system for collecting utilization data for a plurality of accounts in a distributed storage cluster, the system comprising a controller and a plurality of nodes, each node coupled to a plurality of drives, wherein:
each of the nodes includes one or more processors configured to collect storage utilization data for a one account by:
extracting, at a given node, a plurality of storage utilization parameters recorded under the one account from each of a plurality of account databases that are maintained by a plurality of drives coupled to the given node until all drives coupled to the given node are processed;
generating a raw storage data file based on the plurality of storage utilization parameters extracted at the given node to the raw storage data file;
storing the raw storage data file on the given node;
locating, by the given node periodically searching for, the raw storage data file on the given node;
upon the raw storage data file being located:
copying the raw storage data file to the cluster until the raw storage data file is successfully copied to the cluster;
upon successfully copying the raw storage data file to the cluster, marking the raw storage data file as copied;
uploading the raw storage data file to the controller until the raw storage data file is successfully uploaded to the controller; and
upon successfully uploading the raw storage data file to the controller, deleting the raw storage data file.

### Claim 24 (depends on 23)

The system of claim 23, wherein the distributed storage cluster further includes a plurality of proxy servers, each proxy server coupled to one or more nodes to coordinate access requests with its corresponding nodes, wherein:
each proxy server includes one or more processors configured to collect transfer utilization data for the one account by:
creating, at a given proxy server, a rotated log file based on a log file being maintained by the given proxy server, wherein the log file contains entries for each access request processed by the given proxy server;
locating, by the given proxy server periodically searching for, the rotated log file on the given proxy server;
upon the rotated log file being located:
renaming the rotated log file such that the rotated log file becomes a preprocess file;
parsing the preprocess file on the given proxy server to extract raw transfer data, wherein the raw transfer data parsed from the preprocess file are aggregated into aggregated transfer data until the parsing completes;
transferring, from the given proxy server, the raw transfer data to the cluster;
generating, at the given proxy server, an aggregated transfer data file based on the aggregated transfer data;
determining whether both transferring the raw data and generating the aggregated transfer data file are successful;
if both transferring the raw data and generating the aggregated transfer data file are successful, uploading the aggregated transfer data file to the controller until the aggregated transfer data file is successfully uploaded to the controller; and
upon successfully uploading the aggregated transfer data file to the controller, deleting the aggregated transfer data file.

### Claim 25 (depends on 24)

The system of claim 24, wherein the controller includes one or more processors configured to generate a cluster account interval (CAI) data for the one account based on the raw storage data file and the aggregated transfer data file,
wherein each CAI datum is derived from one record from each of the raw storage data file and the aggregated transfer data file, and
wherein the CAI data include storage and transfer utilization data over a predetermined time span.

### Claim 26 (depends on 24)

The system of claim 24, further comprising:
deleting the preprocess file if the aggregated transfer data file is generated successfully.
