Efficient indexing of documents with similar content
AssigneeGoogle Inc.
InventorsJeffrey A. Dean, Sanjay Ghemawat, Gautham Thambidorai
Life of this patent
Estimating the term…
Abstract
A computer system comprising one or more processors and memory groups a set of documents into a plurality of clusters. Each cluster includes one or more documents of the set of documents and a respective cluster of documents of the plurality of clusters includes respective cluster data corresponding to a plurality of documents including a first document and a second document. The computer system determines that the second document includes duplicate data that is duplicative of corresponding data in the first document, identifies a respective subset of the respective cluster data that excludes at least a subset of the duplicate data, and generates an index of the respective subset of the respective cluster data.
Claims29
A method of processing documents, comprising:
- at a computer system having one or more processors and memory storing one or more programs for execution by the one or more processors:
- grouping a set of documents into a plurality of clusters, wherein each cluster includes one or more documents of the set of documents and a respective cluster of documents of the plurality of clusters includes respective cluster data corresponding to a plurality of documents including a first document and a second document;
- determining that the second document includes duplicate data that is duplicative of corresponding data in the first document;
- identifying a respective subset of the respective cluster data that excludes at least a subset of the duplicate data;
- and
- generating an index of the respective subset of the respective cluster data.
A computer system, comprising:
- one or more processors;
- memory;
- and
- one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
- grouping a set of documents into a plurality of clusters, wherein each cluster includes one or more documents of the set of documents and a respective cluster of documents of the plurality of clusters includes respective cluster data corresponding to a plurality of documents including a first document and a second document;
- determining that the second document includes duplicate data that is duplicative of corresponding data in the first document;
- identifying a respective subset of the respective cluster data that excludes the duplicate data;
- and
- generating an index of the respective subset of the respective cluster data.
A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computer system with one or more processors, cause the computer system to:
- group a set of documents into a plurality of clusters, wherein each cluster includes one or more documents of the set of documents and a respective cluster of documents of the plurality of clusters includes respective cluster data corresponding to a plurality of documents including a first document and a second document;
- determine that the second document includes duplicate data that is duplicative of corresponding data in the first document;
- identify a respective subset of the respective cluster data that excludes the duplicate data;
- and
- generate an index of the respective subset of the respective cluster data.
Dependencies are inferred from the claim text.