# US 8,543,551 B2 · Collaborative, distributed, data de-duplication

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 8,543,551 B2
- Document ID: US8543551B2
- Status: Granted
- Kind: B2
- Application number: 13/182,498
- Filed: 2011-07-14
- Published: 2012-01-19
- Granted: 2013-09-24
- Earliest claimed: 2011-07-14
- Applicants: Jeffrey Vincent Tofano
- Inventors: Jeffrey Vincent Tofano
- Web page: https://patentgrep.com/patent/US8543551B2

## CPC

- G06F 7/00: Methods or arrangements for processing data by operating upon the order or content of the data handled (logic circuits H03K19/00)
- G06F 16/1752: {based on file chunks}
- G06F 17/00: Digital computing or data processing equipment or methods, specially adapted for specific functions (information retrieval, database structures or file system structures therefor G06F16/00)
- H03M 7/30: Compression (speech analysis-synthesis for redundancy reduction G10L19/00; for image communication H04N); Expansion; Suppression of unnecessary data, e.g. redundancy reduction
- G06F 3/0641: {De-duplication techniques}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H03M: CODING; DECODING; CODE CONVERSION IN GENERAL (using fluidic means F15C4/00; optical analogue/digital converters G02F7/00; coding, decoding or code conversion, specially adapted for particular applications, see the relevant subclasses, e.g. G01D, G01R, G06F, G06T, G09G, G10L, G11B, G11C, H04B, H04L, H04M, H04N; ciphering or deciphering for cryptography or other purposes involving the need for secrecy G09C)

## Abstract

Example apparatus, methods, and computers participate in collaborative, distributed, data de-duplication. One example method includes initializing a layered parser in a first node in a collaborative distributed data de-duplication (CDDD) topology with a first set of de-duplication control parameters. After transmitting some information to another node in the CDDD topology, the method includes selectively reconfiguring the layered parser in response to feedback acquired from the second node in the CDDD topology. The feedback concerns the data provided by the layered parser.

## Claims

### Claim 1

A method, comprising:
upon determining that a first node configured with a parameterized reactive agent is going to participate in a collaborative, distributed data de-duplication operation in a collaborative, distributed, data de-duplication (CDDD) topology,
controlling the first node, in accordance with a set of parameters associated with the parameterized reactive agent, to create a de-duplication negotiation packet (DNPkt) associated with processing raw data available to the first node;
controlling the first node to transmit the DNPkt to a second node in the CDDD topology;
receiving, from the second node, a reactive control packet (RCPkt) responsive to the DNPkt; and
selectively reconfiguring the parameterized reactive agent by modifying the set of parameters in response to information in the RCPkt.

### Claim 2 (depends on 1)

The method of claim 1, where the parameterized reactive agent is configured to control the first node to perform, in accordance with the set of parameters:
selectively producing a chunk of raw data from the raw data available at the first node,
selectively producing a hash of the chunk of raw data,
selectively indexing the chunk based on the hash,
selectively transmitting the chunk to the second node, and selectively transmitting the hash to the second node.

### Claim 3 (depends on 2)

The method of claim 2, comprising:
controlling the parameterized reactive agent to control the first node to perform, in accordance with the set of parameters:
selectively producing the chunk of raw data,
selectively producing the hash of the chunk of raw data,
selectively indexing the chunk based on the hash,
selectively transmitting the chunk to the second node, and
selectively transmitting the hash to the second node.

### Claim 4 (depends on 1)

The method of claim 1, comprising:
receiving the parameterized reactive agent at the first node.

### Claim 5 (depends on 4)

The method of claim 4, comprising:
establishing an initial set of parameters for a newly received parameterized reactive agent in the first node.

### Claim 6 (depends on 1)

The method of claim 1, where the DNPkt comprises:
a portion of the raw data;
a hash of the portion of the raw data; and
context data associated with the raw data.

### Claim 7 (depends on 6)

The method of claim 6, where the DNPkt comprises:
two or more different portions of the raw data, where the two or more different portions have different lengths; and
hashes associated with the two or more different portions.

### Claim 8 (depends on 6)

The method of claim 6, where the context data identifies one or more of, a file type associated with the raw data, whether the raw data has been previously de-duplicated, a source associated with the raw data, a destination associated with the raw data, whether the data is associated with a replication event, whether the raw data is associated with a rule for parsing data, and whether the raw data is associated with configuring a layered parser.

### Claim 9 (depends on 8)

The method of claim 8, where the set of parameters are determined based, at least in part, on the context data.

### Claim 10 (depends on 8)

The method of claim 8, where the composition of the DNPkt is determined, at least in part, by the context data.

### Claim 11 (depends on 1)

The method of claim 1, where the RCPkt includes one or more of:
information identifying a desired de-duplication node for the raw data;
information identifying a desired de-duplication time for the raw data;
information identifying a desired rule for de-duplicating the raw data;
information identifying a desired chunk size for chunking the raw data;
a statistic from processing data in the DNPkt; and
a request from the second node for a subsequent DNPkt.

### Claim 12 (depends on 1)

The method of claim 1, where determining that the first node is going to participate in the collaborative, distributed, data de-duplication operation is performed on-the-fly, just-in-time, during collaborative de-duplication actions involving two or more nodes in the CDDD topology.

### Claim 13

An apparatus, comprising:
a data store to store:
raw data to be de-duplicated as part of a collaborative, distributed process involving two or more apparatus;
processed data de-duplicated as part of the collaborative, distributed process;
indexing data for identifying and retrieving the processed data; and
configuration information for controlling how the apparatus performs its portion of the collaborative, distributed process;
a processor;
a memory; and
an interface to connect the processor, the memory, and a set of logics, the set of logics comprising:
a chunk logic configured to participate in the collaborative, distributed process by creating a chunk of data from the raw data, where the chunk of data is a subset of the raw data;
a hash logic configured to participate in the collaborative, distributed process by computing a hash for the chunk of data;
an index logic configured to participate in the collaborative, distributed process by selectively storing the chunk of data in the data store and by manipulating the indexing data based, at least in part, on the hash;
a chunk transmission logic configured to transmit the chunk to a second apparatus involved in the collaborative, distributed process;
a hash transmission logic configured to transmit the hash to the second apparatus; and
a control logic configured to control the chunk logic, the hash logic, the index logic, the chunk transmission logic, and the hash transmission logic based on the configuration information and on feedback data from the second apparatus.

### Claim 14 (depends on 13)

The apparatus of claim 13, where the chunk logic selectively produces chunks of different sizes using different chunking criteria based on control information provided by the control logic as determined by the configuration information and the feedback data.

### Claim 15 (depends on 13)

The apparatus of claim 13, where the hash logic selectively produces different types of hashes based on control information provided by the control logic as determined by the configuration information and the feedback data.

### Claim 16 (depends on 13)

The apparatus of claim 13, where the index logic is configured to control the second apparatus to store one or more of, the chunk, and the hash on a data store associated with the second apparatus.

### Claim 17 (depends on 13)

The apparatus of claim 13, where the chunk transmission logic and the hash transmission logic are configured to produce a de-duplication negotiation packet that identifies how the control logic is configured based on the results produced by the chunk logic and the hash logic and to provide the de-duplication negotiation packet to the second apparatus as a solicitation for the feedback data.

### Claim 18 (depends on 13)

The apparatus of claim 13, where the de-duplication negotiation packet comprises:
one or more chunks produced by the chunk logic;
one or more related hashes produced by the hash logic; and
context data associated with the raw data to be de-duplicated,
where the context data identifies one or more of, a file type associated with the raw data, whether the raw data has been previously de-duplicated, a source associated with the raw data, a destination associated with the raw data, whether the data is associated with a replication event, whether the raw data is associated with a rule for parsing data, and whether the raw data is associated with configuring a layered parser.

### Claim 19 (depends on 13)

The apparatus of claim 13, where the feedback data comprises:
information identifying additional nodes to participate in the collaborative, distributed process;
information identifying when the raw data is to be de-duplicated; and
information identifying how the raw data is to be de-duplicated.