# US 2020/0401557 A1 · Metadata compaction in a distributed storage system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0401557 A1
- Document ID: US20200401557A1
- Status: Published application
- Kind: A1
- Application number: 16/828,948
- Filed: 2020-03-24
- Published: 2020-12-24
- Earliest claimed: 2019-06-24
- Assignees: Western Digital Technologies, Inc.
- Applicants: Western Digital Technologies, Inc.
- Inventors: Koen Struyve; Thomas Demoor; Wim Vander Schelden
- Web page: https://patentgrep.com/patent/US20200401557A1

## CPC

- G06F 3/0607: {by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device}
- G06F 16/14: Details of searching files based on file metadata
- G06F 3/0604: {Improving or facilitating administration, e.g. storage management}
- G06F 3/0608: {Saving storage space on storage systems}
- G06F 3/061: {Improving I/O performance}
- G06F 3/0626: {Reducing size or complexity of storage systems}
- G06F 3/064: {Management of blocks}
- G06F 3/0643: {Management of files}
- G06F 3/067: {Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]}
- G06F 3/0673: {Single storage device}
- G06F 16/2228: {Indexing structures}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems and methods for metadata compaction in a distributed storage system with a file system interface are described. A file system interface and an object storage system interface use a metadata index for mapping object identifiers from the object storage system to location identifiers for the file system. When the metadata index includes a number of entries for continuous data blocks with overlapping intervals, a defragmentation operation may generate a defragmented entry for a defragmentation interval overlapping the overlapping data blocks.

## Claims

### Claim 1

A system, comprising:
a file system interface configured to process file data requests for data from a file system client;
an object storage system interface configured to process object storage requests to an object storage system; and
a metadata index manager configured to:
manage a metadata index, wherein:
the metadata index includes a plurality of entries;
each entry of the plurality of entries includes:
an object identifier for the object storage system; and
a file system location identifier for a block of data; and
the plurality of entries includes a plurality of continuous data blocks; and
generate, responsive to a defragmentation operation, a defragmented entry in the metadata index, wherein the defragmented entry includes a defragmentation interval traversing the plurality of continuous data blocks.

### Claim 2 (depends on 1)

The system of claim 1, further comprising:
a compactor configured to:
determine the defragmentation interval for a first determined quantity of the plurality of entries;
read blocks of data for the defragmentation interval from the object storage system; and
write a defragmented block of data for the defragmentation interval in the object storage system, wherein writing the block of data for the defragmentation interval generates the defragmented entry in the metadata index.

### Claim 3 (depends on 2)

The system of claim 2, wherein:
the metadata index manager is further configured to:
add the plurality of entries in sequential order in the metadata index based on a timestamp associated with an operation associated with each entry;
access entries in the metadata index in reverse chronological order for new data operations; and
add the defragmented entry following the plurality of entries for the plurality of continuous data blocks.

### Claim 4 (depends on 2)

The system of claim 2, wherein:
the file system location identifier for each entry of the plurality of entries defines:
a block start location;
a block end location; and
a block length for an interval of the block of data; and
a new file system location identifier for the defragmented entry:
identifies the defragmented block of data spanning the defragmentation interval;
defines a defragmented block start location equal to an earliest block start location among the plurality of continuous data blocks; and
defines a defragmented block end location equal to a latest block end location among the plurality of continuous data blocks.

### Claim 5 (depends on 2)

The system of claim 2, wherein:
the compactor is further configured to:
remove entries in the plurality of entries preceding the defragmented entry that include a block of data within the defragmentation interval.

### Claim 6 (depends on 2)

The system of claim 2, further comprising:
a defragmentation initiator configured to:
determine an operation type for a storage operation; and
initiate, responsive to the operation type being a write operation, the defragmentation operation, wherein the compactor includes a write block for the write operation in the defragmentation interval.

### Claim 7 (depends on 2)

The system of claim 2, further comprising:
a defragmentation initiator configured to:
determine an operation type for a storage operation; and
initiate, responsive to the operation type being a read operation, the defragmentation operation, wherein the compactor includes a read interval for the read operation in the defragmentation interval.

### Claim 8 (depends on 7)

The system of claim 7, wherein the read interval is the defragmentation interval.

### Claim 9 (depends on 1)

The system of claim 1, further comprising:
a defragmentation initiator configured to:
determine a fragmentation value for the metadata index based on a relationship between:
a file size value for the file system client; and
an object size value for the object storage system;
determine a fragmentation threshold based on a performance impact value for the object storage system; and
initiate, responsive to the fragmentation value meeting the fragmentation threshold, the defragmentation operation.

### Claim 10 (depends on 1)

The system of claim 1, further comprising:
a defragmentation initiator configured to:
receive a maintenance operation request for the defragmentation operation; and
initiate, responsive to receiving the maintenance operation request, the defragmentation operation.

### Claim 11

A computer-implemented method, comprising:
processing, through a file system interface, file data requests for data from a file system client;
processing, through an object storage system interface, object storage requests to an object storage system;
managing a metadata index, wherein:
the metadata index includes a plurality of entries;
each entry of the plurality of entries includes:
an object identifier for the object storage system; and
a file system location identifier for a block of data; and
the plurality of entries includes a plurality of continuous data blocks with continuous intervals; and
generating, responsive to a defragmentation operation, a defragmented entry in the metadata index, wherein the defragmented entry includes a defragmentation interval traversing the plurality of continuous data blocks.

### Claim 12 (depends on 11)

The computer-implemented method of claim 11, further comprising:
determining the defragmentation interval for a first determined quantity of the plurality of entries;
reading blocks of data for the defragmentation interval from the object storage system; and
writing a defragmented block of data for the defragmentation interval in the object storage system, wherein writing the defragmented block of data for the defragmentation interval generates the defragmented entry in the metadata index.

### Claim 13 (depends on 12)

The computer-implemented method of claim 12, wherein the managing the metadata index comprises:
adding the plurality of entries in sequential order in the metadata index based on a timestamp associated with an operation associated with each entry;
accessing entries in the metadata index in reverse chronological order for new data operations; and
adding the defragmented entry following the plurality of entries for the plurality of continuous data blocks.

### Claim 14 (depends on 12)

The computer-implemented method of claim 12, wherein:
the file system location identifier for each entry of the plurality of entries defines:
a block start location;
a block end location; and
a block length for an interval of the block of data; and
a new file system location identifier for the defragmented entry:
identifies the defragmented block of data spanning the defragmentation interval;
defines a defragmented block start location equal to an earliest block start location among the plurality of continuous data blocks; and
defines a defragmented block end location equal to a latest block end location among the plurality of continuous data blocks.

### Claim 15 (depends on 12)

The computer-implemented method of claim 12, further comprising:
removing entries in the plurality of entries preceding the defragmented entry that include a block of data within the defragmentation interval.

### Claim 16 (depends on 12)

The computer-implemented method of claim 12, further comprising:
determining an operation type for a storage operation; and
initiating, responsive to the operation type being a write operation, the defragmentation operation, wherein a write block for the write operation is in the defragmentation interval.

### Claim 17 (depends on 12)

The computer-implemented method of claim 12, further comprising:
determining an operation type for a storage operation; and
initiating, responsive to the operation type being a read operation, the defragmentation operation, wherein a read interval for the read operation is in the defragmentation interval.

### Claim 18 (depends on 11)

The computer-implemented method of claim 11, further comprising:
determining a fragmentation value for the metadata index based on a relationship between:
a file size value for the file system client; and
an object size value for the object storage system;
determining a fragmentation threshold based on a performance impact value for the object storage system; and
initiating, responsive to the fragmentation value meeting the fragmentation threshold, the defragmentation operation.

### Claim 19 (depends on 11)

The computer-implemented method of claim 11, further comprising:
receiving a maintenance operation request for the defragmentation operation; and
initiating, responsive to receiving the maintenance operation request, the defragmentation operation.

### Claim 20

A system, comprising:
means for processing, through a file system interface, file data requests for data from a file system client;
means for processing, through an object storage system interface, object storage requests to an object storage system;
means for managing a metadata index, wherein:
the metadata index includes a plurality of entries;
each entry of the plurality of entries includes:
an object identifier for the object storage system; and
a file system interface location identifier for a block of data; and
the plurality of entries includes a plurality of continuous data blocks with continuous intervals; and
means for generating, responsive to a defragmentation operation, a defragmented entry in the metadata index, wherein the defragmented entry includes a defragmentation interval traversing the plurality of continuous data blocks.