US 11,886,446 B2Granted

Cross-lingual language models and pretraining of cross-lingual language models

AssigneeBaidu USA LLC

InventorsHongliang Fei, Puxuan Yu, Ping Li

Life of this patent

Estimating the term…

Loading patent…

Abstract

Existing research on cross-lingual retrieval cannot take good advantage of large-scale pretrained language models, such as multilingual BERT and XLM. The absence of cross-lingual passage-level relevance data for finetuning and the lack of query-document style pretraining are some of the key factors of this issue. Accordingly, embodiments of two novel retrieval-oriented pretraining tasks are presented herein to further pretrain cross-lingual language models for downstream retrieval tasks, such as cross-lingual ad-hoc retrieval (CUR) and cross-lingual question answering (CLQA). In one or more embodiments, distant supervision data was constructed from multilingual texts using section alignment to support retrieval-oriented language model pretraining. In one or more embodiments, directly finetuning language models on part of an evaluation collection was performed by making Transformers capable of accepting longer sequences. Experiments show that model embodiments significantly improve upon general multilingual language models in at least the cross-lingual retrieval setting and the cross-lingual transfer setting.

Claims20

Dependent claims

A computer-implemented method for pretraining a cross-lingual language model, the method comprising:

responsive to a stop condition not being reached, performing steps comprising:
given a relevance ranking batch of cross-lingual query and document pairs, in which the relevance ranking batch comprises (1) at least one positive sample cross-lingual query and document pair, in which the query and the document are in different languages and the document is relevant to the query and (2) at least one negative sample cross-lingual query and document pair, in which the query and the document are in different languages and the document is not relevant to the query, using the cross-lingual language model with a relevance ranking model head to predict a relevance ranking that ranks whether a document is relevant to a corresponding query;
updating the cross-lingual language model using known relevance rankings and predicted relevance rankings;
given a query language modeling batch of cross-lingual query and document pairs, in which the query and the document are in different languages and the document is relevant to the query, inputting the query, in which a portion of tokens in the query are masked tokens, and the corresponding document into the cross-lingual language model to predict the masked tokens of the query;
and
updating the cross-lingual language model based upon the cross-lingual language model's prediction compared to the actual masked tokens;
and
responsive to a stop condition being reached, outputting the cross-lingual language model, which has been updated, as a pretrained cross-lingual language model.

5 more elements

A system comprising:

one or more processors;
and
a non-transitory computer-readable medium or media comprising one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:
responsive to a stop condition not being reached, performing steps comprising:
given a relevance ranking batch of cross-lingual query and document pairs, in which the relevance ranking batch comprises (1) at least one positive sample cross-lingual query and document pair, in which the query and the document are in different languages and the document is relevant to the query and (2) at least one negative sample cross-lingual query and document pair, in which the query and the document are in different languages and the document is not relevant to the query, using a cross-lingual language model with a relevance ranking model head to predict a relevance ranking that ranks whether a document is relevant to a corresponding query;
updating the cross-lingual language model using known relevance rankings and predicted relevance rankings;
given a query language modeling batch of cross-lingual query and document pairs, in which the query and the document are in different languages and the document is relevant to the query, inputting the query, in which a portion of tokens in the query are masked tokens, and the corresponding document into the cross-lingual language model to predict the masked tokens of the query;
and
updating the cross-lingual language model based upon the cross-lingual language model's prediction compared to the masked tokens;
and
responsive to a stop condition being reached, outputting the cross-lingual language model, which has been updated, as a pretrained cross-lingual language model.

8 more elements

A computer-implemented method for pretraining a cross-lingual language model, the method comprising:

using a pretrained multilingual language representation model to initialize a cross-lingual language model that uses global-plus-sliding window attention, in which a global window for attention is used for query tokens and a sliding window for attention is used for at least some document tokens;
responsive to a stop condition not being reached, performing steps comprising:
given cross-lingual query and document pairs in which at least some of query tokens of a query input into the cross-lingual language model are masked, training the cross-lingual language model using a query language model objective regarding predicting, for a query, the masked query tokens given a corresponding relevant cross-lingual document;
and
given cross-lingual query and document pairs, training the cross-lingual language model with a relevance ranking model head to predict a relevance ranking that ranks whether a document is relevant to a corresponding query;
and
responsive to a stop condition being reached, outputting the cross-lingual language model, which has been updated, as a pretrained cross-lingual language model.

4 more elements

Dependencies are inferred from the claim text.

Facts

Application: 17/575,650Filed 14 Jan 2022 · us provisional application
Priority: 5 Apr 2021
Published: US 2022/0318255 A16 Oct 2022
Granted: 30 Jan 2024
Term ends: Not estimatedThe source records don’t support an estimate.

Classification5 codes

G06F 16/24578
With adaptation to user needs: using ranking
G06F 16/3337
Query translation: translation of the query language, e.g. Chinese to English
G06F 40/284
Recognition of textual entities: lexical analysis, e.g. tokenisation or collocates
G06F 40/58
Processing or translation of natural language: use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for…