Back to results
US 11,238,222 B2Granted
Dual monolingual cross-entropy-delta filtering of noisy parallel data
AssigneeBeijing Didi Infinity Technology and Development Co., Ltd.
InventorAmittai Axelrod
Life of this patent
Estimating the term…
Abstract
Sentence pairs can be filtered using dual monolingual cross-entropy-delta filtering. The filtered sentence pairs can be used to train a model for performing a language processing task, such as machine translation.
Claims17
A non-transitory computer-readable storage medium for filtering sentence pairs in two languages, the storage medium storing instructions, when executed by one or more processors, cause the one or more processors to perform operations comprising:
- receiving a corpus of sentence pairs, wherein a first half and a second half of each of the corpus of sentence pairs are associated with a first label of a first language and a second label of a second language, respectively;
- determining a plurality of feature scores for each of the plurality of sentence pairs using:
- dual monolingual cross-entropy deltas of (i) the first half of the sentence pair, with respect to a plurality of first training sentences, according to a first monolingual language model of the first language and a second monolingual language model of the first language and (ii) the second half of the sentence pair, with respect to a plurality of second training sentences, according to a first monolingual language model of the second language and a second monolingual language model of the second language, and
- a first monolingual rank score for the first half of the sentence pair determined using a first ranking of the first half of the sentence pair when selecting the sentence pair to train a first language model that models representative first sentences in the first language, and a second monolingual rank score for the second half of the sentence pair determined using a second ranking of the second half of the sentence pair when selecting the sentence pair to train a second language model, corresponding to the first language model, that models representative second sentences in the second language;
- determining an overall score for each of the plurality of sentence pairs using the plurality of feature scores for the sentence pair;
- selecting a subcorpus of sentence pairs of the corpus of sentence pairs for performing a language processing task, wherein the selecting is based on the overall score of each of the plurality of sentence pairs and a predetermined threshold of a number of different words in the first language comprised in the first halves of all sentence pairs of the subcorpus of sentence pairs.
3 more elements
A system for filtering sentence pairs in two languages comprising:
- one or more processors;
- and
- a memory storing instructions that, when executed by the one or more processors, cause the system to perform:
- receiving a plurality of sentence pairs, wherein a first half and a second half of each of the plurality of sentence pairs are associated with a first label of a first language and a second label of a second language, respectively;
- determining a plurality of feature scores for each of the plurality of sentence pairs according to a first monolingual language model of the first language, a second monolingual language model of the first language, a first monolingual language model of the second language, and a second monolingual language model of the second language, wherein the determining comprises:
- determining a dual monolingual cross-entropy delta-score for each of the plurality of sentence pairs using, dual monolingual cross-entropy deltas with respect to the first half and the second half of the sentence air according to the first monolingual language model of the first language, the second monolingual language model of the first language, the first monolingual language model of the second language, and the second monolingual language model of the second language, and
- determining a first monolingual rank score and a second monolingual rank score for the first half and the second halt respectively, of the sentence pair determined using a first ranking of the first half of the sentence pair and a second ranking of the second half of the sentence pair, respectively, when selecting the first half and the second half of the sentence pair to train a first language model and a second language model, respectively, that model representative first sentences in the first language and second sentences in the second language, respectively;
- determining an overall score for each of the plurality of sentence pairs using the plurality of feature scores for the sentence pair;
- and
- selecting a subset of sentence pairs of the plurality of sentence pairs using the overall score of each of the plurality of sentence pairs for performing a language processing task using the subset of sentence pairs, wherein the selecting is based on the overall score of each of the plurality of sentence pairs and a predetermined threshold of a number of different words in the first language comprised in the first halves of all sentence pairs of the subcorpus of sentence pairs.
7 more elements
Dependencies are inferred from the claim text.