System and method for duplicate text recognition

A non-transitory computer readable media having stored thereon data representing a sequence of instructions for duplicate text recognition, the sequence of instructions which, when executed by a processor, cause the processor to perform:

• (a) dividing an electronic text into a plurality of phrase segments;
• (b) converting each of the phrase segments into a unique and fixed-length bit string;
• (c) storing in a search engine a plurality of groups of the bit strings, each group of bit strings (string group) comprising a plurality of bit strings respectively corresponding to the phrase segments in a particular electronic text;
• (d) determining whether a predefined similarity between any two string groups in the search engine reaches a first threshold;
• (e) determining the two electronic texts corresponding to the two string groups are duplicate texts if the predefined similarity between the two string groups reaches the first threshold;
• and (f) determining the two electronic texts corresponding to the two string groups are not duplicate texts if the predefined similarity between the two string groups is less than the first threshold.

A system for duplicate text recognition comprising:

• a segmentation unit for dividing an electronic text into a plurality of phrase segments;
• a conversion unit connected with the segmentation unit and configured for converting each of the phrase segments into a unique and fixed-length bit string;
• a search engine connected with the conversion unit and configured for storing a plurality of groups of the bit strings, each group of bit strings (string group) comprising a plurality of bit strings respectively corresponding to the phrase segments in a particular electronic text;
• and a judgment unit connected with the search engine, the judgment unit being configured for determining whether a predefined similarity between any two string groups in the search engine reaches a first threshold, for determining the two electronic texts corresponding to the two string groups are duplicate texts if the predefined similarity between the two string groups reaches the first threshold, and for determining the two electronic texts corresponding to the two string groups are not duplicate texts if the predefined similarity between the two string groups is less than the first threshold.