Text deduplication method and apparatus, and storage medium

A text deduplication apparatus, comprising:

• a processor and a memory connected to the processor, the memory storing a machine readable instruction executable to the processor, and the processor executing the machine readable instruction to perform:
• obtaining a text set, the text set comprising a plurality of to-be-deduplicated texts;
• capturing, for each to-be-deduplicated text, a corresponding subtext string from the to-be-deduplicated text, by:
• capturing, for each to-be-deduplicated text, one or more subtext strings from the to-be-deduplicated text, to obtain a text string set corresponding to the to-be-deduplicated text;
• determining, in the text set, to-be-deduplicated texts having a same subtext string, to obtain text subsets, wherein each subtext string corresponds to a text subset, and each text subset includes one or more to-be-deduplicated texts that have the corresponding subtext string, and the text subsets are determined by:
• establishing, for each to-be-deduplicated text, an inverted index between the to-be-deduplicated text and each of the one or more subtext strings in the text string set corresponding to the to-be-deduplicated text;
• and
• determining, based on the inverted index, the to-be-deduplicated texts having the same subtext string in the text set, to obtain the text subset corresponding to each subtext string;
• performing text deduplication processing on the text subset corresponding to each subtext string, to obtain a deduplicated text set corresponding to each subtext string;
• and
• obtaining, according to the deduplicated text set corresponding to each subtext string, a result text set of the text set after the deduplication.

A non-transitory computer readable storage medium, the storage medium storing a machine readable instruction, and the machine readable instruction being executable to a processor to perform:

• obtaining a text set, the text set comprising a plurality of to-be-deduplicated texts;
• capturing, for each to-be-deduplicated text, a corresponding subtext string from the to-be-deduplicated text, by:
• capturing, for each to-be-deduplicated text, one or more subtext strings from the to-be-deduplicated text, to obtain a text string set corresponding to the to-be-deduplicated text;
• determining, in the text set, to-be-deduplicated texts having a same subtext string, to obtain text subsets, wherein each subtext string corresponds to a text subset, and each text subset includes one or more to-be-deduplicated texts that have the corresponding subtext string, and the text subsets are determined by:
• establishing, for each to-be-deduplicated text, an inverted index between the to-be-deduplicated text and each of the one or more subtext strings in the text string set corresponding to the to-be-deduplicated text;
• and
• determining, based on the inverted index, the to-be-deduplicated texts having the same subtext string in the text set, to obtain the text subset corresponding to each subtext string;
• performing text deduplication processing on the text subset corresponding to each subtext string, to obtain a deduplicated text set corresponding to each subtext string;
• and
• obtaining, according to the deduplicated text set corresponding to each subtext string, a result text set of the text set after the deduplication.