System And Method for Filtering Datasets Using Conditional-Likelihood Filtration

A system for generating a filtered dataset for training a language processing model, the system comprising:

• a processor;
• a memory in communication with the processor, the memory comprising computer executable instructions that when executed by the processor cause the processor to:
• obtain an unfiltered dataset including a plurality of documents;
• obtain one or more trigger phrases;
• determine a conditional likelihood score for each trigger phrase conditioned on each document in the unfiltered dataset, the conditional likelihood score being based on contextual content of the respective document;
• and
• generate the filtered dataset for training the language processing model by removing one or more documents which have the conditional likelihood score exceeding a predefined threshold.

A non-transitory computer readable medium for generating a filtered dataset for training a language processing model, the computer readable medium comprising computer executable instructions to:

• obtain an unfiltered dataset including a plurality of documents;
• obtain one or more trigger phrases;
• determine a conditional likelihood score for each trigger phrase conditioned on each document in the unfiltered dataset, the conditional likelihood score being based on contextual content of the respective document;
• and
• generate the filtered dataset for training the language processing model by removing one or more documents which have the conditional likelihood score exceeding a predefined threshold.