# US 2014/0156687 A1 · Identifying duplicate files

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2014/0156687 A1
- Document ID: US20140156687A1
- Status: Published application
- Kind: A1
- Application number: 13/149,548
- Filed: 2011-05-31
- Published: 2014-06-05
- Earliest claimed: 2011-05-31
- Inventors: Qiliang Chen
- Web page: https://patentgrep.com/patent/US20140156687A1

## CPC

- G06F 16/1748: {De-duplication implemented within the file system, e.g. based on file segments (de-duplication techniques in storage systems for the management of data blocks G06F3/0641)}
- G06F 16/9558: {Details of hyperlinks; Management of linked annotations}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, are provided for identifying duplicate data items. A method is provided that include the actions of identifying a file to be downloaded; determining that the size of the file exceeds a threshold value; in response to determining that the size of the file exceeds the threshold value, requesting one or more portions of the file, the one or more portions being less than the entire file; upon receiving the requested one or more portions of the file, constructing a file signature for the file based on the one or more portions of the file; determining whether the constructed signature exists in a file signature repository; if a matching file signature exists in the file signature repository, recording location information for the file to be downloaded as associated with the matching file signature.

## Claims

### Claim 1

A method, comprising:
identifying a downloadable file from a first network location;
determining that the size of the file exceeds a threshold value;
in response to determining that the size of the file exceeds the threshold value, determining whether the file is a duplicate of a previously stored file from a second network location, the determining comprising:
requesting one or more portions of the file from the first network location, the one or more portions being less than the entire file;
upon receiving the requested one or more portions of the file, constructing a file signature for the file based on the one or more portions of the file;
determining whether to download the complete file including determining whether the constructed signature constructed from the one or more received portions of the file exists in a file signature repository; and
if it is determined that the constructed file signature matches a file signature that exists in the file signature repository, the file signature being associated with the stored file, determining that the downloadable file is a duplicate file of the stored file, wherein as a result of the determination, the duplicate file is not downloaded and location information for the duplicate file at the first network location is recorded as associated with the matching file signature of the stored file.

### Claim 2 (depends on 1)

The method of claim 1, where determining that the size of the file exceeds the threshold value includes requesting a header of the file and comparing the size of the file identified in the header with the threshold value.

### Claim 3 (depends on 1)

The method in claim 1, wherein the file signature repository maintains a list of file signatures, their corresponding files, and their respective file sizes.

### Claim 4 (depends on 3)

The method in claim 3, wherein the file signature repository maintains a list of file signatures, links to their corresponding files, and their respective file sizes.

### Claim 5 (depends on 4)

The method of claim 4, further comprising: recording the name and location of the file to be downloaded in association with the matching file signature in the file signature repository.

### Claim 6 (depends on 1)

The method of claim 1, wherein the one or more portions of the file include a beginning portion of the file having a specified size.

### Claim 7 (depends on 1)

The method of claim 1, wherein the one or more portions of the file are each of a fixed size and are located at specified noncontiguous locations in the file.

### Claim 8 (depends on 1)

The method of claim 1, wherein constructing a file signature for the file comprises:
combining the one or more portions of the file into a whole binary string as the file signature.

### Claim 9 (depends on 1)

The method of claim 1, wherein constructing a file signature for the file comprises:
combining the one or more portions of the file into a whole binary string; and
hashing on the binary string to obtain a binary value of a predetermined length as the file signature.

### Claim 10 (depends on 1)

The method of claims 1, wherein determining whether the constructed signature exists in a file signature repository comprises:
identifying entries in the file signature repository with the same file size as the file to be downloaded; and
comparing the constructed file signature of the file to be downloaded with the file signature in each identified entry from the file signature repository to determine if there is a match.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
if there is no matching file signature already existing in the file signature repository, downloading the file in full; and
recording the constructed file signature in the file signature repository as associated with the file.

### Claim 12

A method, comprising:
identifying in a web page a link to a downloadable file from a first network location, the file having a size over a threshold value;
determining whether the downloadable file is a duplicate of a previously stored file from a second network location, the determining comprising:
requesting one or more noncontiguous portions of the downloadable file from the first network location without receiving one or more additional portions of the downloadable file;
constructing a file signature for the downloadable file using the one or more portions of the file;
determining whether the file signature already exists in a signature repository, wherein determining whether the file signature already exists in the signature repository includes comparing the constructed file signature with existing signatures in the signature repository;
if it is determined that the file signature exists in the signature repository, the downloadable file is determined to be a duplicate of the stored file and in response to the determination that the downloadable file is a duplicate, the file is not downloaded and location information for the downloadable file at the first network location is recorded as associated with the matching file signature of the stored file; and
if it is determined that the file signature does not exist in the signature repository, the file is determined not to be a duplicate of the stored file, wherein in response to the determination that the downloadable file is not a duplicate:
downloading the file and storing the downloaded file in a file storage; and
storing the file signature in the file signature repository, wherein the file signature is associated with a link to the downloaded file in the file storage and the first network location from which the file was downloaded.

### Claim 13

A system comprising:
one or more computing devices configured to perform operations comprising:
identifying a downloadable file from a first network location;
determining that the size of the file exceeds a threshold value;
in response to determining that the size of the file exceeds the threshold value, determining whether the file is a duplicate of a previously stored file from a second network location, the determining comprising:
requesting one or more portions of the file from the first network location, the one or more portions being less than the entire file;
upon receiving the requested one or more portions of the file, constructing a file signature for the file based on the one or more portions of the file;
determining whether to download the complete file including determining whether the constructed signature constructed from the one or more received portions of the file exists in a file signature repository; and
if it is determined that the constructed file signature matches a file signature that exists in the file signature repository, the file signature being associated with the stored file, determining that the downloadable file is a duplicate file of the stored file, wherein as a result of the determination, the duplicate file is not downloaded and location information for the duplicate file at the first network location is recorded as associated with the matching file signature of the stored file.

### Claim 14 (depends on 13)

The system of claim 13, where determining that the size of the file exceeds the threshold value includes requesting a header of the file and comparing the size of the file identified in the header with the threshold value.

### Claim 15 (depends on 13)

The system in claim 13, wherein the file signature repository maintains a list of file signatures, their corresponding files, and their respective file sizes.

### Claim 16 (depends on 15)

The system in claim 15, wherein the file signature repository maintains a list of file signatures, links to their corresponding files, and their respective file sizes.

### Claim 17 (depends on 16)

The system of claim 16, further configured to perform operations comprising: recording the name and location of the file to be downloaded in association with the matching file signature in the file signature repository.

### Claim 18 (depends on 13)

The system of claim 13, wherein the one or more portions of the file include a beginning portion of the file having a specified size.

### Claim 19 (depends on 13)

The system of claim 13, wherein the one or more portions of the file are each of a fixed size and are located at specified noncontiguous locations in the file.

### Claim 20 (depends on 13)

The system of claim 13, wherein constructing a file signature for the file comprises:
combining the one or more portions of the file into a whole binary string as the file signature.

### Claim 21 (depends on 13)

The system of claim 13, wherein constructing a file signature for the file comprises:
combining the one or more portions of the file into a whole binary string; and
hashing on the binary string to obtain a binary value of a predetermined length as the file signature.

### Claim 22 (depends on 13)

The system of claims 13, wherein determining whether the constructed signature exists in a file signature repository comprises:
identifying entries in the file signature repository with the same file size as the file to be downloaded; and
comparing the constructed file signature of the file to be downloaded with the file signature in each identified entry from the file signature repository to determine if there is a match.

### Claim 23 (depends on 13)

The system of claim 13, further configured to perform operations comprising:
if there is no matching file signature already existing in the file signature repository, downloading the file in full; and
recording the constructed file signature in the file signature repository as associated with the file.

### Claim 24

A system comprising:
one or more computing devices configured to perform operations comprising:
identifying in a web page a link to a downloadable file from a network location, the file having a size over a threshold value;
determining whether the downloadable file is a duplicate of a previously stored file from a second network location, the determining comprising:
requesting one or more noncontiguous portions of the downloadable file from the first network location without receiving one or more additional portions of the downloadable file;
constructing a file signature for the downloadable file using the one or more portions of the file;
determining whether the file signature already exists in a signature repository, wherein determining whether the file signature already exists in the signature repository includes comparing the constructed file signature with existing signatures in the signature repository;
if it is determined that the file signature exists in the signature repository, the downloadable file is determined to be a duplicate of the stored file and in response to the determination that the downloadable file is a duplicate, the file is not downloaded and location information for the downloadable file at the first network location is recorded as associated with the matching file signature of the stored file; and
if it is determined that the file signature does not exist in the signature repository, the file is determined not to be a duplicate of the stored file, wherein in response to the determination that the downloadable file is not a duplicate:
downloading the file and storing the downloaded file in a file storage; and
storing the file signature in the file signature repository, wherein the file signature is associated with a link to the downloaded file in the file storage and the first network location from which the file was downloaded.

### Claim 25

A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:
identifying a downloadable file from a first network location;
determining that the size of the file exceeds a threshold value;
in response to determining that the size of the file exceeds the threshold value, determining whether the file is a duplicate of a previously stored file from a second network location, the determining comprising:
requesting one or more portions of the file from the first network location, the one or more portions being less than the entire file;
upon receiving the requested one or more portions of the file, constructing a file signature for the file based on the one or more portions of the file;
determining whether to download the complete file including determining whether the constructed signature constructed from the one or more received portions of the file exists in a file signature repository; and
if it is determined that the constructed file signature matches a file signature that exists in the file signature repository, the file signature being associated with the stored file, determining that the downloadable file is a duplicate file of the stored file, wherein as a result of the determination, the duplicate file is not downloaded and location information for the duplicate file at the first network location is recorded as associated with the matching file signature of the stored file.

### Claim 26 (depends on 25)

The computer storage medium of claim 25, where determining that the size of the file exceeds the threshold value includes requesting a header of the file and comparing the size of the file identified in the header with the threshold value.

### Claim 27 (depends on 25)

The computer storage medium in claim 25, wherein the file signature repository maintains a list of file signatures, their corresponding files, and their respective file sizes.

### Claim 28 (depends on 27)

The computer storage medium in claim 27, wherein the file signature repository maintains a list of file signatures, links to their corresponding files, and their respective file sizes.

### Claim 29 (depends on 28)

The computer storage medium of claim 28, further comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising: recording the name and location of the file to be downloaded in association with the matching file signature in the file signature repository.

### Claim 30 (depends on 25)

The computer storage medium of claim 25, wherein the one or more portions of the file include a beginning portion of the file having a specified size.

### Claim 31 (depends on 25)

The computer storage medium of claim 25, wherein the one or more portions of the file are each of a fixed size and are located at specified noncontiguous locations in the file.

### Claim 32 (depends on 25)

The computer storage medium of claim 25, wherein constructing a file signature for the file comprises:
combining the one or more portions of the file into a whole binary string as the file signature.

### Claim 33 (depends on 25)

The computer storage medium of claim 25, wherein constructing a file signature for the file comprises:
combining the one or more portions of the file into a whole binary string; and
hashing on the binary string to obtain a binary value of a predetermined length as the file signature.

### Claim 34 (depends on 25)

The computer storage medium of claims 25, wherein determining whether the constructed signature exists in a file signature repository comprises:
identifying entries in the file signature repository with the same file size as the file to be downloaded; and
comparing the constructed file signature of the file to be downloaded with the file signature in each identified entry from the file signature repository to determine if there is a match.

### Claim 35 (depends on 25)

The computer storage medium of claim 25, further comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:
if there is no matching file signature already existing in the file signature repository, downloading the file in full; and
recording the constructed file signature in the file signature repository as associated with the file.

### Claim 36

A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:
identifying in a web page a link to a downloadable file from a network location, the file having a size over a threshold value;
determining whether the downloadable file is a duplicate of a previously stored file from a second network location, the determining comprising:
requesting one or more noncontiguous portions of the downloadable file from the first network location without receiving one or more additional portions of the downloadable file;
constructing a file signature for the downloadable file using the one or more portions of the file;
determining whether the file signature already exists in a signature repository, wherein determining whether the file signature already exists in the signature repository includes comparing the constructed file signature with existing signatures in the signature repository;
if it is determined that the file signature exists in the signature repository, the downloadable file is determined to be a duplicate of the stored file and in response to the determination that the downloadable file is a duplicate, the file is not downloaded and location information for the downloadable file at the first network location is recorded as associated with the matching file signature of the stored file; and
if it is determined that the file signature does not exist in the signature repository, the file is determined not to be a duplicate of the stored file, wherein in response to the determination that the downloadable file is not a duplicate:
downloading the file and storing the downloaded file in a file storage; and
storing the file signature in the file signature repository, wherein the file signature is associated with a link to the downloaded file in the file storage and the first network location from which the file was downloaded.