System and method for document analysis, processing and information extraction

A method for improving a measure of similarity between elements in a dataset, comprising the steps of:

• receiving a dataset comprising N digital documents;
• computing by a processor a diffusion geometry of said dataset comprising at least a plurality of diffusion coordinates, said coordinates being scaled by diffusion time and admitting an extension to any additional new digital document, wherein the time scale represents the length of inference paths between documents;
• storing first k diffusion coordinates for each digital document in the dataset comprising N digital documents in a memory, wherein k depends on a selected time scale and selected precision;
• adding new digital documents to original dataset by calculating and storing said extension of said first k diffusion coordinates by said processor;
• and
• calculating and storing in said memory an approximation of a diffusion distance between all pairs of digital documents in said dataset by said processor, said approximation being calculated by applying the Euclidean metric to k stored diffusion coordinates of said all pairs of digital documents, and wherein precision of said approximation is related to the chosen time scale and a value of said k;
• and
• using said approximate diffusion distance as a new measure of similarity between all pairs of digital documents in said dataset.

A computer system configured to improving a measure of similarity between elements in a dataset comprising N digital documents, comprising:

• a processor for computing a diffusion geometry of said dataset comprising at least a plurality of diffusion coordinates, said coordinates being scaled by diffusion time and admitting an extension to any additional new digital document;
• wherein a time scale represents a length of inference paths between documents;
• a memory for storing first k diffusion coordinates for each digital document in said dataset comprising N digital documents, wherein k depends on selected time scale and selected precision;
• and wherein said processor is operable to add new digital documents to said original dataset by calculating and storing extension of said first k diffusion coordinates;
• wherein said processor is operable to calculate and store an approximation of the diffusion distance between all pairs of digital documents in said dataset in said memory, said approximation being calculated by applying the Euclidean metric to said k stored diffusion coordinates of said pairs of digital documents, and where the precision of said approximation is related to the chosen time scale and a value of said k;
• and
• wherein said processor is operable to use said approximate diffusion distance as a new measure of similarity between all pairs of digital documents in said dataset.