Identifying similarities within large collections of unstructured data
AssigneeVerdasys, Inc.
InventorMichael Smolsky
Life of this patent
Estimating the term…
Abstract
A technique for determining when documents stored in digital format in a data processing system are similar. A method compares a sparse representation of two or more documents by breaking the documents into “chunks” of data of predefined sizes. Selected subsets of the chunks are determined as being representative of data in the documents and coefficients are developed to represent such chunks. Coefficients are then combined into coefficient clusters containing coefficients that are similar according to a predetermined similarity metric. The degree of similarity between documents is then evaluated by counting clusters into which chunks of similar documents fall.
Claims16
A method for determining if a first and second document stored in a digital format in a data processing system are similar by comparing sparse representations of the two documents, the method comprising the steps of:
- breaking the first and second documents into chunks of data of predefined sizes;
- selecting a subset of all chunks as representative of the data in the document;
- determining a set of coefficients that represent selected chunks;
- combining sets of coefficients into coefficient clusters, a coefficient cluster containing coefficients which are similar according to a predetermined similarity metric;
- and
- evaluating a degree of similarity between documents by counting clusters into which chunks from both documents fall.
Dependencies are inferred from the claim text.