# US 10,769,106 B2 · System and method for identifying and mitigating redundancies in stored data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,769,106 B2
- Document ID: US10769106B2
- Status: Granted
- Kind: B2
- Application number: 15/653,001
- Filed: 2017-07-18
- Published: 2017-11-02
- Granted: 2020-09-08
- Earliest claimed: 2006-08-18
- Assignees: FalconStor, Inc.
- Applicants: FalconStor, Inc.
- Inventors: Wai Lam
- Web page: https://patentgrep.com/patent/US10769106B2

## CPC

- G06F 16/13: File access structures, e.g. distributed indices (arrangements of input from, or output to, record carriers G06F3/06)
- G06F 16/137: {Hash-based (content-based indexing of textual data G06F16/31)}
- G06F 16/174: Redundancy elimination performed by the file system (error detection or correction of the data by redundancy in operations G06F11/14)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

In an example of an embodiment of the invention, a repeating pattern is identified within stored data comprising a plurality of data files, each data file comprising at least a header section and a data section stored in an unknown format. At least one occurrence of the repeating pattern is identified as a header section of a respective data file, and a data section of the respective data file is identified based, at least in part, on a location of the at least one occurrence of the repeating pattern. The identified data section of the respective data file is backed up. Systems are also disclosed.

## Claims

### Claim 1

A method to manage data by one or more processing devices, the method comprising:
identifying a repeating pattern in data stored in a storage system, by one of the one or more processing devices, the data comprising a plurality of pairs of header sections and data sections, wherein:
a format of the header sections in the storage system are unknown to or cannot be processed by the processing device, and locations of the header sections are unknown to the processing device; and
a format of the data sections are unknown to or cannot be processed by the processing device, and locations of the data sections are unknown to the processing device;
associating the repeating pattern with the header sections;
determining locations of the header sections in the plurality of pairs in the storage system based on the association;
determining the locations of respective data sections in the plurality of pairs, in the storage system, based, at least in part, on the determined locations of respective header sections; and
performing a selected data processing operation on the data sections in the determined locations by the one or more processing devices.

### Claim 2 (depends on 1)

The method of claim 1, wherein the stored data is maintained on a tape storage medium.

### Claim 3 (depends on 1)

The method of claim 1, wherein the data processing operation comprises backing up the data sections.

### Claim 4 (depends on 3)

The method of claim 3, wherein backing up the second components comprises:
generating a first digest representing a respective data section;
comparing the first digest to a set of second digests: and
if the first digest is found in the set of second digests, replacing the respective data section in the stored data by the first digest.

### Claim 5 (depends on 4)

The method of claim 4, further comprising, if the first digest is not found in the set of second digests:
adding the first digest to the set of second digests;
backing up the identified data section; and
replacing the identified data section in the stored data by the first digest.

### Claim 6 (depends on 1)

The method of claim 1, wherein the repeating pattern indicates at least one characteristic of the header section chosen from the group consisting of: a location of the header section, a structure of the header section, a length of the header section, a beginning of the header section, and an end of the header section.

### Claim 7 (depends on 1)

The method of claim 1, wherein the repeating pattern comprises a sequence of values that repeats identically within the data.

### Claim 8

A system for performing data processing operations on stored data, comprising:
storage configured to store data; and
one or more processing devices configured to:
identify a repeating pattern in data stored in a storage system the data comprising a plurality of pairs of header sections and data sections, wherein:
a format of the header sections in the storage system are unknown to or cannot be processed by the processing device, and locations of the header sections are unknown to the processing device; and
a format of the data sections are unknown to or cannot be processed by the processing device, and locations of the data sections are unknown to the processing device;
determine locations of the header sections in the plurality of pairs, in the storage system, based on the association;
determine the locations of respective data sections in the plurality of pairs, in the storage system, based, at least in part, on the determined locations of respective data sections; and
perform a selected data processing operation on the data sections in the determined locations.

### Claim 9 (depends on 8)

The system of claim 8, wherein the stored data is maintained on a tape storage medium.

### Claim 10 (depends on 9)

The system of claim 9, wherein the one or more processing devices is further configured to:
back up the identified data section in a second storage different from the first storage.

### Claim 11 (depends on 9)

The system of claim 9, wherein the first storage system comprises a virtual tape library.

### Claim 12 (depends on 8)

The system of claim 8, wherein the data processing operation comprises backing up the data sections.

### Claim 13 (depends on 12)

The system of claim 12, wherein the one or more processing devices is configured to back up the second components by:
generating a first digest representing a respective data section;
comparing the first digest to a set of second digests; and
if the first digest is found in the set of second digests, replacing the respective data section in the stored data by the first digest.

### Claim 14 (depends on 13)

The system of claim 13, wherein the one or more processing devices is further configured to, if the first digest is not found in the set of second digests:
add the first digest to the set of second digests;
back up the identified data section; and
replace the identified data section in the stored data by the first digest.

### Claim 15 (depends on 8)

The system of claim 8, wherein the repeating pattern indicates at least in part, on at least one characteristic of the header section chosen from the group consisting of: a location of the header section, a structure of the header section, a length of the header section, a beginning of the header section, and an end of the header section.

### Claim 16 (depends on 8)

The system of claim 8, wherein the repeating pattern comprises a sequence of values that repeats identically within the data.

### Claim 17 (depends on 8)

The system of claim 8, wherein one of the one or more processing devices is further configured to:
identify a format of the header section based, at least in part, on the repeating pattern.

### Claim 18 (depends on 8)

The system of claim 8, wherein each occurrence of the repeating pattern comprises a sequence chosen from the group consisting of: a sequence of bits, and a sequence of bytes.