Identifying duplicate files

A method, comprising:

• identifying in a web page a link to a downloadable file from a first network location, the file having a size over a threshold value;
• determining whether the downloadable file is a duplicate of a previously stored file from a second network location, the determining comprising:
• requesting one or more noncontiguous portions of the downloadable file from the first network location without receiving one or more additional portions of the downloadable file;
• constructing a file signature for the downloadable file using the one or more portions of the file;
• determining whether the file signature already exists in a signature repository, wherein determining whether the file signature already exists in the signature repository includes comparing the constructed file signature with existing signatures in the signature repository;
• if it is determined that the file signature exists in the signature repository, the downloadable file is determined to be a duplicate of the stored file and in response to the determination that the downloadable file is a duplicate, the file is not downloaded and location information for the downloadable file at the first network location is recorded as associated with the matching file signature of the stored file;
• and
• if it is determined that the file signature does not exist in the signature repository, the file is determined not to be a duplicate of the stored file, wherein in response to the determination that the downloadable file is not a duplicate:
• downloading the file and storing the downloaded file in a file storage;
• and
• storing the file signature in the file signature repository, wherein the file signature is associated with a link to the downloaded file in the file storage and the first network location from which the file was downloaded.

A system comprising:

• one or more computing devices configured to perform operations comprising:
• identifying a downloadable file from a first network location;
• determining that the size of the file exceeds a threshold value;
• in response to determining that the size of the file exceeds the threshold value, determining whether the file is a duplicate of a previously stored file from a second network location, the determining comprising:
• requesting one or more portions of the file from the first network location, the one or more portions being less than the entire file;
• upon receiving the requested one or more portions of the file, constructing a file signature for the file based on the one or more portions of the file;
• determining whether to download the complete file including determining whether the constructed signature constructed from the one or more received portions of the file exists in a file signature repository;
• and
• if it is determined that the constructed file signature matches a file signature that exists in the file signature repository, the file signature being associated with the stored file, determining that the downloadable file is a duplicate file of the stored file, wherein as a result of the determination, the duplicate file is not downloaded and location information for the duplicate file at the first network location is recorded as associated with the matching file signature of the stored file.

A system comprising:

• one or more computing devices configured to perform operations comprising:
• identifying in a web page a link to a downloadable file from a network location, the file having a size over a threshold value;
• determining whether the downloadable file is a duplicate of a previously stored file from a second network location, the determining comprising:
• requesting one or more noncontiguous portions of the downloadable file from the first network location without receiving one or more additional portions of the downloadable file;
• constructing a file signature for the downloadable file using the one or more portions of the file;
• determining whether the file signature already exists in a signature repository, wherein determining whether the file signature already exists in the signature repository includes comparing the constructed file signature with existing signatures in the signature repository;
• if it is determined that the file signature exists in the signature repository, the downloadable file is determined to be a duplicate of the stored file and in response to the determination that the downloadable file is a duplicate, the file is not downloaded and location information for the downloadable file at the first network location is recorded as associated with the matching file signature of the stored file;
• and
• if it is determined that the file signature does not exist in the signature repository, the file is determined not to be a duplicate of the stored file, wherein in response to the determination that the downloadable file is not a duplicate:
• downloading the file and storing the downloaded file in a file storage;
• and
• storing the file signature in the file signature repository, wherein the file signature is associated with a link to the downloaded file in the file storage and the first network location from which the file was downloaded.

A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

• identifying a downloadable file from a first network location;
• determining that the size of the file exceeds a threshold value;
• in response to determining that the size of the file exceeds the threshold value, determining whether the file is a duplicate of a previously stored file from a second network location, the determining comprising:
• requesting one or more portions of the file from the first network location, the one or more portions being less than the entire file;
• upon receiving the requested one or more portions of the file, constructing a file signature for the file based on the one or more portions of the file;
• determining whether to download the complete file including determining whether the constructed signature constructed from the one or more received portions of the file exists in a file signature repository;
• and
• if it is determined that the constructed file signature matches a file signature that exists in the file signature repository, the file signature being associated with the stored file, determining that the downloadable file is a duplicate file of the stored file, wherein as a result of the determination, the duplicate file is not downloaded and location information for the duplicate file at the first network location is recorded as associated with the matching file signature of the stored file.

A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

• identifying in a web page a link to a downloadable file from a network location, the file having a size over a threshold value;
• determining whether the downloadable file is a duplicate of a previously stored file from a second network location, the determining comprising:
• requesting one or more noncontiguous portions of the downloadable file from the first network location without receiving one or more additional portions of the downloadable file;
• constructing a file signature for the downloadable file using the one or more portions of the file;
• determining whether the file signature already exists in a signature repository, wherein determining whether the file signature already exists in the signature repository includes comparing the constructed file signature with existing signatures in the signature repository;
• if it is determined that the file signature exists in the signature repository, the downloadable file is determined to be a duplicate of the stored file and in response to the determination that the downloadable file is a duplicate, the file is not downloaded and location information for the downloadable file at the first network location is recorded as associated with the matching file signature of the stored file;
• and
• if it is determined that the file signature does not exist in the signature repository, the file is determined not to be a duplicate of the stored file, wherein in response to the determination that the downloadable file is not a duplicate:
• downloading the file and storing the downloaded file in a file storage;
• and
• storing the file signature in the file signature repository, wherein the file signature is associated with a link to the downloaded file in the file storage and the first network location from which the file was downloaded.