# US 2018/0025046 A1 · Reference Set Construction for Data Deduplication

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0025046 A1
- Document ID: US20180025046A1
- Status: Published application
- Kind: A1
- Application number: 15/604,567
- Filed: 2017-05-24
- Published: 2018-01-25
- Earliest claimed: 2016-07-19
- Assignees: Western Digital Technologies, Inc.
- Applicants: Western Digital Technologies, Inc.
- Inventors: Ashish Singhai; Ashwin Narasimha; Vijay Karamcheti; Tanay Goel
- Web page: https://patentgrep.com/patent/US20180025046A1

## CPC

- G06F 16/1748: {De-duplication implemented within the file system, e.g. based on file segments (de-duplication techniques in storage systems for the management of data blocks G06F3/0641)}
- G06F 3/0608: {Saving storage space on storage systems}
- G06F 3/0641: {De-duplication techniques}
- G06F 3/0652: {Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket}
- G06F 3/0683: {Plurality of storage devices}
- G06F 11/3034: {where the computing system component is a storage system, e.g. DASD based or network based (digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097)}
- G06F 11/3409: {for performance assessment}
- G06F 16/1724: {Details of de-fragmentation performed by the file system (saving storage space on storage systems G06F3/0608; management of blocks in storage devices G06F3/064)}
- G06F 16/1727: {Details of free space management performed by the file system (saving storage space on storage systems G06F3/0608; management of blocks in storage devices G06F3/064)}
- G06F 16/1752: {based on file chunks}
- G06F 16/215: Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
- G06F 16/2291: {User-Defined Types; Storage management thereof}
- G06F 3/0604: {Improving or facilitating administration, e.g. storage management}
- G06F 11/1453: {using de-duplication of the data}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

By way of example, a data storage system may comprise, a non-transitory storage device storing data blocks in chunks, and a storage logic coupled to the non-transitory storage device that manages storage of data on the storage device. The storage logic is executable to receive a data stream for storage in a non-transitory storage device, the data stream including one or more data blocks, analyze the data stream to determine a domain, retrieve a pre-configured reference set based on the domain, and deduplicate the one or more data blocks of the data stream using the pre-configured reference set.

## Claims

### Claim 1

A data storage system, comprising:
a non-transitory storage device storing data blocks in chunks;
a storage logic coupled to the non-transitory storage device that manages storage of data on the non-transitory storage device, the storage logic executable to:
receive a data stream for storage in the non-transitory storage device, the data stream including one or more data blocks;
analyze the data stream to determine a domain;
retrieve a pre-configured reference set based on the domain; and
deduplicate the one or more data blocks of the data stream using the pre-configured reference set.

### Claim 2 (depends on 1)

The data storage system of claim 1, wherein the storage logic is further executable to:
evaluate performance of the pre-configured reference set in deduplicating data blocks of one or more prior data streams;
assign a performance score to the pre-configured reference set based on the evaluated performance; and
generate a ranking for the pre-configured reference set for the domain relative to other reference sets associated with the domain based on the performance score, wherein retrieving the pre-configured reference set includes retrieving the pre-configured reference set based further on the ranking of the pre-configured reference set.

### Claim 3 (depends on 2)

The data storage system of claim 2, wherein the performance of the pre-configured reference set is evaluated based on a deduplication ratio.

### Claim 4 (depends on 2)

The data storage system of claim 2, wherein the performance of the pre-configured reference set is evaluated based on a deduplication speed.

### Claim 5 (depends on 1)

The data storage system of claim 1, wherein the domain is a range of physical area of the non-transitory storage device that hosts data blocks with similar characteristics.

### Claim 6 (depends on 1)

The data storage system of claim 1, wherein the storage logic is further executable to:
generate the pre-configured reference set as a synthetic reference set for data block sequences that repeat at a rate above a predetermined threshold;
identify a duplicated data block sequence;
retrieve the synthetic reference set based on the duplicated data block sequence; and
deduplicate the identified duplicated data block sequence using the synthetic reference set.

### Claim 7 (depends on 1)

The data storage system of claim 1, wherein the storage logic is further executable to:
assign a use count variable to the pre-configured reference set;
update the use count variable when the pre-configured reference set is used to deduplicate the data blocks; and
store the pre-configured reference set including the use count variable in the non-transitory storage device.

### Claim 8

A method comprising:
receiving a data stream for storage in a non-transitory storage device, the data stream including one or more data blocks;
analyzing the data stream to determine a domain;
retrieving a pre-configured reference set based on the domain; and
deduplicating the one or more data blocks of the data stream using the pre-configured reference set.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
evaluating performance of the pre-configured reference set in deduplicating data blocks of one or more prior data streams;
assigning a performance score to the pre-configured reference set based on the evaluated performance; and
generating a ranking for the pre-configured reference set for the domain relative to other reference sets associated with the domain based on the performance score, wherein retrieving the pre-configured reference set includes retrieving the pre-configured reference set based further on the ranking of the pre-configured reference set.

### Claim 10 (depends on 9)

The method of claim 9, wherein the performance of the pre-configured reference set is evaluated based on a deduplication ratio.

### Claim 11 (depends on 9)

The method of claim 9, wherein the performance of the pre-configured reference set is evaluated based on a deduplication speed.

### Claim 12 (depends on 8)

The method of claim 8, wherein the domain is a range of physical area of the non-transitory storage device that hosts data blocks with similar characteristics.

### Claim 13 (depends on 8)

The method of claim 8, further comprising:
generating the pre-configured reference set as a synthetic reference set for data block sequences that repeat at a rate above a predetermined threshold;
identifying a duplicated data block sequence;
retrieving the synthetic reference set based on the duplicated data block sequence; and
deduplicating the identified duplicated data block sequence using the synthetic reference set.

### Claim 14 (depends on 8)

The method of claim 8, further comprising:
assigning a use count variable to the pre-configured reference set;
updating the use count variable when the pre-configured reference set is used to deduplicate the data blocks; and
storing the pre-configured reference set including the use count variable in the non-transitory storage device.

### Claim 15

A data storage system comprising:
means for receiving a data stream for storage in a non-transitory storage device, the data stream including one or more data blocks;
means for analyzing the data stream to determine a domain;
means for retrieving a pre-configured reference set based on the domain; and
means for deduplicating the one or more data blocks of the data stream using the pre-configured reference set.

### Claim 16 (depends on 15)

The data storage system of claim 15, further comprising:
means for evaluating performance of the pre-configured reference set in deduplicating data blocks of one or more prior data streams;
means for assigning a performance score to the pre-configured reference set based on the evaluated performance; and
means for generating a ranking for the pre-configured reference set for the domain relative to other reference sets associated with the domain based on the performance score, wherein retrieving the pre-configured reference set includes retrieving the pre-configured reference set based further on the ranking of the pre-configured reference set.

### Claim 17 (depends on 16)

The data storage system of claim 16, wherein the performance of the pre-configured reference set is evaluated based on a deduplication ratio.

### Claim 18 (depends on 16)

The data storage system of claim 16, wherein the performance of the pre-configured reference set is evaluated based on a deduplication speed.

### Claim 19 (depends on 15)

The data storage system of claim 15, wherein the domain is a range of physical area of the non-transitory storage device that hosts data blocks with similar characteristics.

### Claim 20 (depends on 15)

The data storage system of claim 15, further comprising:
means for generating the pre-configured reference set as a synthetic reference set for data block sequences that repeat at a rate above a predetermined threshold;
means for identifying a duplicated data block sequence;
means for retrieving the synthetic reference set based on the duplicated data block sequence; and
means for deduplicating the identified duplicated data block sequence using the synthetic reference set.
