# US 2021/0174033 A1 · Method and apparatus for evaluating translation quality > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0174033 A1 - Document ID: US20210174033A1 - Status: Published application - Kind: A1 - Application number: 16/894,974 - Filed: 2020-06-08 - Published: 2021-06-10 - Earliest claimed: 2019-12-05 - Assignees: Baidu Online Network Technology (Beijing) Co., Ltd. - Applicants: Baidu Online Network Technology (Beijing) Co., Ltd. - Inventors: Hao Xiong; Ruiqing Zhang; Junjie Li; Zhongjun He; Zhi Li; Hua Wu; Haifeng Wang - Web page: https://patentgrep.com/patent/US20210174033A1 ## CPC - G06F 40/51: Translation evaluation - G06F 40/289: Phrasal analysis, e.g. finite state techniques or chunking - G06F 40/30: Semantic analysis - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Embodiments of the present disclosure relate to a method and apparatus for evaluating a translation quality. The method may include: acquiring a to-be-evaluated translation and a reference translation; inputting the to-be-evaluated translation and the reference translation into a pre-trained repetition coding model to obtain a semantic similarity between the to-be-evaluated translation and the reference translation, the repetition coding model being a neural network for calculating a probability of a pair of sentences being repetition sentences; analyzing the to-be-evaluated translation and the reference translation into two syntax trees respectively; calculating a similarity between the two syntax trees as a text similarity between the to-be-evaluated translation and the reference translation; and using a weighted sum of the semantic similarity and the text similarity as a translation quality score. ## Claims ### Claim 1 A method for evaluating a translation quality, comprising: acquiring a to-be-evaluated translation and a reference translation; inputting the to-be-evaluated translation and the reference translation into a pre-trained repetition coding model to obtain a semantic similarity between the to-be-evaluated translation and the reference translation, the repetition coding model being a neural network for calculating a probability of a pair of sentences being repetition sentences; analyzing the to-be-evaluated translation and the reference translation into two syntax trees respectively; calculating a similarity between the two syntax trees as a text similarity between the to-be-evaluated translation and the reference translation; and using a weighted sum of the semantic similarity and the text similarity as a translation quality score. ### Claim 2 (depends on 1) The method according to claim 1, further comprising: inputting the to-be-evaluated translation into a pre-trained automatic question-and-answer model to obtain at least one group of questions and answers, the automatic question-and-answer model being a neural network for extracting a question and an answer in a text; determining a proportion of correct answers in the at least one group of questions and answers as a question-and-answer score; acquiring a standard score obtained by performing, by the automatic question-and-answer model, an answering result analysis on a standard translation; correcting the question-and-answer score through the standard score to obtain an intelligibility score of the to-be-evaluated translation; and using a weighted sum of the semantic similarity, the text similarity and the intelligibility score as the translation quality score. ### Claim 3 (depends on 1) The method according to claim 1, further comprising: recognizing key information from the to-be-evaluated translation; comparing the recognized key information with manually annotated key information, and calculating a comprehensive score of an accuracy rate and a recall rate of the key information; and correcting the translation quality score through the comprehensive score. ### Claim 4 (depends on 2) The method according to claim 2, wherein the correcting the question-and-answer score through the standard score comprises: setting a predetermined constant to smooth a difference between the standard score and the question-and-answer score. ### Claim 5 (depends on 1) The method according to claim 1, wherein the repetition coding model is trained by: acquiring a set of original sentences of a first language identical to a language of the to-be-evaluated translation; for each original sentence of the first language in the set of the original sentences of the first language, translating the original sentence of the first language into a translation of a second language through a first translation model, translating the translation of the second language into a repetition sentence of the first language through a second translation model, constituting a repetition sentence pair using the original sentence of the first language and the repetition sentence of the first language, and selecting randomly one sentence to constitute a non-repetition sentence pair using the one sentence and the original sentence of the first language; and using a set of repetition sentence pairs as positive samples and the non-repetition sentence pair as a negative sample, and training a classifier using a machine learning method to obtain the repetition coding model. ### Claim 6 (depends on 5) The method according to claim 5, further comprising: controlling a size of a vocabulary table of the first translation model and a size of a vocabulary table of the second translation model, to generate a variety of repetition sentence pairs. ### Claim 7 (depends on 5) The method according to claim 5, wherein a training sample set of the repetition coding model is different from a training sample set of the first translation model, and is also different from a training sample set of the second translation model. ### Claim 8 (depends on 5) The method according to claim 5, wherein in response to determining that the to-be-evaluated translation is in a predetermined mainstream language, a BERT model is used as the repetition coding model. ### Claim 9 An apparatus for evaluating a translation quality, comprising: at least one processor; and a memory storing instructions, the instructions when executed by the at least one processor, causing the at least one processor to perform operations, the operations comprising: acquiring a to-be-evaluated translation and a reference translation; inputting the to-be-evaluated translation and the reference translation into a pre-trained repetition coding model to obtain a semantic similarity between the to-be-evaluated translation and the reference translation, the repetition coding model being a neural network for calculating a probability of a pair of sentences being repetition sentences; analyzing the to-be-evaluated translation and the reference translation into two syntax trees respectively; calculating a similarity between the two syntax trees as a text similarity between the to-be-evaluated translation and the reference translation; and using a weighted sum of the semantic similarity and the text similarity as a translation quality score. ### Claim 10 (depends on 9) The apparatus according to claim 9, the operations further comprising: inputting the to-be-evaluated translation into a pre-trained automatic question-and-answer model to obtain at least one group of questions and answers, the automatic question-and-answer model being a neural network for extracting a question and an answer in a text; determining a proportion of correct answers in the at least one group of questions and answers as a question-and-answer score; acquiring a standard score obtained by performing, by the automatic question-and-answer model, an answering result analysis on a standard translation; correcting the question-and-answer score through the standard score to obtain an intelligibility score of the to-be-evaluated translation; and using a weighted sum of the semantic similarity, the text similarity and the intelligibility score as the translation quality score. ### Claim 11 (depends on 9) The apparatus according to claim 9, the operations further comprising: recognizing key information from the to-be-evaluated translation; comparing the recognized key information with manually annotated key information, and calculate a comprehensive score of an accuracy rate and a recall rate of the key information; and correcting the translation quality score through the comprehensive score. ### Claim 12 (depends on 10) The apparatus according to claim 10, wherein the correcting the question-and-answer score through the standard score comprises: setting a predetermined constant to smooth a difference between the standard score and the question-and-answer score. ### Claim 13 (depends on 9) The apparatus according to claim 9, wherein the repetition coding model is trained by: acquiring a set of original sentences of a first language identical to a language of the to-be-evaluated translation; for each original sentence of the first language in the set of the original sentences of the first language, translating the original sentence of the first language into a translation of a second language through a first translation model, translating the translation of the second language into a repetition sentence of the first language through a second translation model, constituting a repetition sentence pair using the original sentence of the first language and the repetition sentence of the first language, and selecting randomly one sentence to constitute a non-repetition sentence pair using the one sentence and the original sentence of the first language; and using a set of repetition sentence pairs as positive samples and the non-repetition sentence pair as a negative sample, and training a classifier using a machine learning apparatus to obtain the repetition coding model. ### Claim 14 (depends on 13) The apparatus according to claim 13, wherein the operations further comprising: controlling a size of a vocabulary table of the first translation model and a size of a vocabulary table of the second translation model, to generate a variety of repetition sentence pairs. ### Claim 15 (depends on 13) The apparatus according to claim 13, wherein a training sample set of the repetition coding model is different from a training sample set of the first translation model, and is also different from a training sample set of the second translation model. ### Claim 16 (depends on 13) The apparatus according to claim 13, wherein in response to determining that the to-be-evaluated translation is in a predetermined mainstream language, a BERT model is used as the repetition coding model. ### Claim 17 A non-transitory computer readable medium, storing a computer program, wherein the program, when executed by a processor, causes the processor to perform operations, the operations comprising: acquiring a to-be-evaluated translation and a reference translation; inputting the to-be-evaluated translation and the reference translation into a pre-trained repetition coding model to obtain a semantic similarity between the to-be-evaluated translation and the reference translation, the repetition coding model being a neural network for calculating a probability of a pair of sentences being repetition sentences; analyzing the to-be-evaluated translation and the reference translation into two syntax trees respectively; calculating a similarity between the two syntax trees as a text similarity between the to-be-evaluated translation and the reference translation; and using a weighted sum of the semantic similarity and the text similarity as a translation quality score.