Method and system for exemplar learning for templatizing documents across data sources

A method for exemplar learning for templatizing documents across data sources comprising:

• scanning, by a scanner operatively coupled to a processing subsystem, a plurality of files to retrieve a plurality of textual content, wherein the plurality of files are positive files and negative files accommodating one of a similar template and a dissimilar template;
• refining, by a pre-processor module operatively coupled to the scanner, an efficacy corresponding to the scanned plurality of files by using a classifier trained with positive files and negative files using artificial intelligence technique, wherein the artificial intelligence technique includes at least one of a Deep Neural Network (DNN), Convolutional Neural Network (CNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Deep Q-Networks, or a combination thereof, to identify textual content with a similar template, upon scanning;
• extracting, by an extraction module operatively coupled to the pre-processor module, a plurality of common sequences and sub-sequences from the plurality of files, upon pre-processing, wherein the common sequences and sub-sequences are defined by similarity calculations based on a match between a plurality of objects pertaining to the plurality of files, wherein the match is one of an exact match for determining if the plurality of objects are identical and a partial match for determining if the plurality of objects are partially similar;
• applying cryptographic hash techniques to each sequence and performing a match on the hash values, and for a partial match using an advanced hashing scheme including locally sensitive hashing (LSH) for hashing the sequences into buckets so that objects in each of the buckets are similar with a high probability;
• ranking, by a ranking module operatively coupled to the extraction module, the plurality of common sequences and sub-sequences, upon extraction, based on a score wherein the score defines the occurrence of each of the plurality of common sequences and sub-sequences;
• generating, by a feature vector generating module operatively coupled to the ranking module, a feature vector from the plurality of common sequences and sub-sequences, upon ranking, wherein the score is based on a score function wherein the score function is applied to each of the plurality of sequences thereby ranking the said plurality of sequences;
• generating, by the feature vector generating module operatively coupled to the ranking module, a feature vector from the plurality of common sequences and sub-sequences, upon ranking, wherein the feature vector is generated using sentence embeddings including at least one of fastText, Bidirectional Encoder Representations from Transformers (BERT), or Universal Sentence Encoder, or an autoencoder, a hashing scheme, or a combination thereof;
• and
• determining, by a determining module operatively coupled to the feature vector generating module, a representation vector of each document based on the feature vector, wherein the representation vector is a predetermined number of dimensional vectors representing a plurality of features of the documents;
• a similarity score between the representation vector and the feature vector using at least one of cosine similarity, Jaccard similarity, normalized L1 distance, or Manhattan distance;
• and a threshold value for the classifier by building a soft margin classifier (support vector machine) to find critical examples and designing the threshold to maximize the margin between positive and negative examples thereby developing the classifier automatically to search for one or more positive files with similar templates in an organization.