Neural ranking model for generating sparse representations for information retrieval

A computer-implemented method for information retrieval, the method comprising:

• generating, by a document encoder comprising a pretrained language model layer, a sparse representation for each of one or more received documents predicting term importance of the document over a vocabulary;
• generating, by a query encoder, a representation of a received query over the vocabulary;
• comparing the generated representation of the query to the generated representations of the one or more documents to generate a set of respective document scores;
• and
• ranking the one or more documents based on the generated set of document scores;
• wherein the document encoder and the query encoder are respectively separate encoders.

A computer-implemented method for information retrieval, the method comprising:

• generating, by a document encoder comprising a pretrained language model layer, a sparse representation for each of one or more received documents predicting term importance of the document over a vocabulary;
• generating, by a query encoder, a representation of a received query over the vocabulary;
• comparing the generated representation of the query to the generated representations of the one or more documents to generate a set of respective document scores;
• and
• ranking the one or more documents based on the generated set of document scores;
• wherein the pretrained language model of the document encoder is trained by middle training before the language model is fine-tuned for information retrieval.

A computer-implemented method for training a neural ranker of an information retrieval model, the method comprising:

• initializing parameters of the neural ranker;
• providing a dataset comprising documents and queries to a document encoder and a query encoder of the ranker, the document encoder comprising a pretrained language model layer and being configured to receive the documents and generate a sparse representation for each of the documents predicting term importance of the document over a vocabulary, the query encoder being separate from the document encoder and configured to receive the queries and generate a representation of the query over the vocabulary;
• and
• optimizing a loss including a ranking loss based on the generated representations of the one or more documents and queries and at least one regularization loss;
• wherein the ranking loss and/or the at least one regularization loss is weighted by a weighting parameter.

A computer-implemented method for training an encoder, the method comprising:

• middle training a pretrained language model of the encoder;
• and
• fine-tuning the pretrained language model of the encoder for information retrieval after said middle training;
• wherein the encoder after said fine-tuning is configured for generating a sparse representation for each of one or more received documents predicting term importance of the document over a vocabulary.